Oracle Waives E-Business Suite Extended Support Fees, again...

Yesterday, Steven Chan announced on his blog (if you don't read it, you should...) that Oracle was waiving Extended Support fees for E-Business Suite 11i and 12.0. According to the new announcement, the entire Extended Support period (for 11i and 12.0) is now free (if you've already paid for it, contact your salesperson). I haven't dug deeply enough into the history of this, but I don't think that Oracle has done anything to move the actual dates.

This means that customers who are on 11i and 12.0 (who have also met the minimum baseline patch requirements) have some extra time to get to the next release (which is currently 12.1.3).

What is Premier Support?

Premier Support is the “normal” support category that Oracle puts around “current” software. Under Premier Support, Oracle obligates themselves to actually fixing bugs and finding solutions to problems.

According to Oracle's website:

Premier Support – Delivers full system support for your Oracle hardware, operating systems and applications with an upfront, minimum five-year support commitment that helps you plan and budget.

What is Extended Support?

Extended Support is just like Premier Support, except that they charge you more for it. All of the features of Premier Support are there, and they will still produce new bugfixes when they're needed.

According to Oracle's website:

Extended Support – Offers an additional three years of support for select Oracle software and operating systems for an additional fee so you can effectively manage your upgrade strategy.

What comes after Extended Support?

Sustaining Support. Here, you still have access to the support site and analysts. However, no new bugfixes will be produced. If you encounter a previously unknown problem, your only choice may be to upgrade.

According to Oracle's website:

Sustaining Support – Provides investment protection by further extending support for Oracle software, operating systems and select hardware products. Features include access to online support tools, knowledgebases, pre-existing fixes, and assistance from Oracle's technical support experts.

More information is available at: http://www.oracle.com/us/support/lifetime-support/index.html

What does this mean for me, by version?

For Release 11i, Premier Support ended November 30, 2010 and Extended Supprt will now end on November 30, 2013. You'll want to be on 11.5.10.2 with the minimum baseline patches applied (according to Note: 883202.1). Ideally, you should also be on RDBMS 11.2.0.3.

For Release 12.0, Premier Support ended on January 31, 2012 and Extended Support will now end on January 31, 2015. The document you'll need to follow is 1195034.1. For that you will need to be on 12.0.6 (12.0 RUP 6), with the mainimum baseline patches applied (according to the document). Ideally, you should also be on RDBMS 11.2.0.3.

For Release 12.1, Premier Support will end in June of 2014 (I'm not certain, but I believe that it is May 31, 2014) and Extended Support will end on the same date in 2017. Oracle has also (back in October) announced that they are waiving the first year of Extended Support fees for R12.1 (which means you are good until May of 2015). As far as baseline patching is concerned, keep an eye on 1195034.1. At this point, the only minimum baseline requirement for Extended Support is that you have applied at least the R12.1.3 Release Update Pack. As with the other releases, you should also be on RDBMS 11.2.0.3.

Why are they doing this?

Obviously, I have no real information on what's going on inside of Oracle. What I can say is that, based on customers I (and others) have spoken with, there are still a large number of customers on 11i and 12.0. Release 12.2 has been “coming soon” for quite some time, and, with R12.1.3 dates appearing on the horizon, many customers are waiting for R12.2 to be released. They don't want to finish one upgrade project only to immediately start another.

My advice? It is my understanding that you will not be able to upgrade directly from 11.5.10.2 to R12.2 (I could be wrong on this). If you're on 11i, you should be working on your upgrade to R12.1.3 now. If you're on R12.0, you may want to wait until R12.2 comes out and figure out if you can go straight to R12.2. If you're on R12.1, get to R12.1.3 and be ready to start planning your R12.2 upgrade shortly after it is released.

In all cases, get your databases upgraded to 11.2.0.3 as well. Many of the deadlines for 10gR2 have already passed.

– James

Password-less Login Using SSH Pre-Shared Keys

Way back when I started working with Unix (otherwise known as "the olden days" or "days of yore"), one of the tricks we used was a concept known as "remote login" and the "Berkeley R commands". This was based on a number of things, most of them depending on either the /etc/hosts.equiv or the ${HOME}/.rhosts file to establish the trusting relationship. Configuring these would allow you the ability to do some really neat things. Among them, copying files from one host to another using a command like rcp /tmp/file user@remotehost:/tmp/file without being asked for a password. This made for some really neat scripting opportunities and made it much easier to manage multiple systems.

Unfortunately, the Berkeley "R" commands are notoriously insecure. The way that the trusting was done was based entirely on the username and hostname of the remote user on the remote host. Literally, you told the server to trust "jmorrow@remotehost.mydomain.com". The problem with this is that all that was required was knowledge of the trusting relationship. All you had to do was set up a machine named "remotehost.mydomain.com" and create a "jmorrow" user on it. Then you could go anywhere that that trusting relationship allowed.

Fortunately for us, the cool features that were introduced by the Berkeley "R" commands are implemented much more securely in the SSH protocol and toolset.

The SSH Protocol can use pre-shared keys to establish trusting relationships. In this case, each node has both a public and a private key. When the client talks to the server, the client offers a " key". The server, which maintains a list of trusted "public keys", then compares that key to it's database to determine if it actually trusts the client. If the client passes the test, then it is allowed in without any further challenge. This can be very useful for administrators, automated file transfer, also for scripting interactions between hosts. Note that this is not a "Machine A" trusts "Machine B" relationship. It is "user@machinea" trusts "user@machineb".

For the purposes of this article, the "server" is the node that you are logging into from the "client". So, the "server" is the one that is doing the trusting. The terms "server" and "client" refer only to the role being played by each component in the ssh communications session. I should also mention that Oracle Real Application Clusters (RAC) depends on this relationship as well.

Generate your public/private key pairs [Both Client and Server]

The server (user@host) needs to have one, and each client (user@host) that is being trusted needs to have one.

Execute these two commands (in a Unix/Linux environment) to create both your rsa and your dsa keys. You will be prompted for a location to store the files (typically under ${HOME}/.ssh), and for a passphrase. In all cases, it's ok to accept the defaults.

ssh-keygen -t rsa

ssh-keygen -t dsa

If you know you don't want to use a passphrase, you could generate the keys with these two commands:

ssh-keygen -t rsa -f ${HOME}/.ssh/id_rsa -N ""

ssh-keygen -t dsa -f ${HOME}/.ssh/id_dsa -N ""

Transfer the public key files from the client to the server

I prefer to make sure that I have a uniquely named copy of the public keys (makes it easier to transfer to another box when first establishing the relationship).

cd ${HOME}/.ssh

ls -1 id_[dr]sa.pub |while read LINE

do

cp ${LINE} ${LINE}.`whoami`@`hostname -s`

done

Now copy these files to the server:

scp ${LINE}.`whoami`@`hostname -s` trustinguser@trustingserver:.ssh/.

Copy the public keys you're trusting into the authorized_keys file

Here, we'll need to put those keys into the authorized_keys file. Do this for each of the files that you transferred in the previous step.

cd ${HOME}/.ssh

cat >> authorized_keys

Make sure permissions are correct

If the permissions on these files are too open, the trusting relationship will not work. Here are my recommendations:

chmod 600 ${HOME}/.ssh/auth*

chmod 700 ${HOME}/.ssh

chmod 644 ${HOME}/.ssh/id_[dr]sa.pub*

chmod 600 ${HOME}/.ssh/id_[dr]sa

Now, you should be able to ssh from the client to the server witout being prompted for a password.

– James

When Can I Merge E-Business Suite Patches?

This is a question that pops up frequently. In earlier releases of E-Business Suite, the rule was that you should never merge a patch with any of it's prerequisite patches. However, many years ago (I believe it was with AD_PF.E, but I could be mistaken), Oracle introduced the ability to merge a patch and it's prerequisite patches together. There are still a few (small) restrictions.

From the Oracle Applications Maintenance Procedures manual for Release 11.5.10.2 (Part No. B19299-01, pages 3-23 and 3-24):

In general, you can safely merge any Oracle Applications patch with any other Oracle Applications patch. Older split driver patches can be merged with newer unified driver patches. Patches can and should be merged with their listed prerequisite patches to make patch application easier.

However, patches that affect the Applications DBA (AD) product must be handled separately. AD patches can be merged with other AD patches, but AD patches and non-AD patches cannot be merged because AD patches may change the AutoPatch utility itself. Merged AD patches must be created separately and applied before you apply non-AD patches.

I believe you'll also find a similar statement in the 11.5.9 version of the document.

The primary logic here is this: If a patch says it requires a certain AD patch (say, AD.I.7), then what it's really telling you is that there is likely something in the structure of the patch itself that an earlier AD release wouldn't know how to properly handle. Keep in mind that AD patches are, generally speaking, patches to the patching tool.

Now, I have seen some patches that warn against merging. And, given the nature of ATG patches, you may be wise to exercise a level of caution and apply them independently as well.

The long and the short of it is, if you're reasonably current (certainly 11.5.10.2 and newer), it is generally safe to merge a patch and it's prerequisite together. Just pay close attention to the READMEs, and watch out for those AD patches.

– James

Skipping the Oracle Configuration Manager prompts in OPatch

One of the more interesting things Oracle has done in recent years is the introduction of Oracle Configuration Manager. Oracle Configuration Manager is a support tool that provides a way to bundle information about your environment in order to help resolve SRs.

However, many customers are concerned with the information that this tool might reveal, so they tend to avoid providing the contact information that OCM requires. As a result, when you run OPatch, you're constantly bugged for that information. Here's how you can get around that:

Generate a response file (either provide the information, or leave things blank):

${ORACLE_HOME}/OPatch/ocm/bin/emocmrsp -output ${HOME}/ocm.rsp

OCM Installation Response Generator 10.3.1.2.0 - Production

Copyright (c) 2005, 2009, Oracle and/or its affiliates. All rights reserved.

Provide your email address to be informed of security issues, install and

initiate Oracle Configuration Manager. Easier for you if you use your My

Oracle Support Email address/User Name.

Visit http://www.oracle.com/support/policies.html for details.

Email address/User Name:

You have not provided an email address for notification of security issues.

Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y

The OCM configuration response file (/home/oracle/ocm.rsp) was successfully created.

Now, when you apply the patch, point opatch to that OCM response file:

opatch -ocmrf ${HOME}/ocm.rsp

– James

Spreadsheet Risk (and why ad-hoc reporting tools make me twitchy)

First, let me say that I'm a DBA, not an accountant. We tend to trust databases to hold and organize data. We use applications and reports developed by professional developers to retrieve that data. Those applications and reports go through a software development lifecycle for a reason: to make sure that they are accurate.

Despite this, many professional developers aren't writing "well-tuned code". They're generally happy to get the right results and, as long as it isn't painfully slow, performance is either an afterthought or the DBA's problem. I've got news for you... some 80% of performance issues are caused by poorly tuned code!  

This is not to denigrate developers.  I'm saying this mostly to prove a point:  if you can't reliably expect well-tuned code from a professional developer, you're insane if you expect anything better from end-users with an ad-hoc query tool.

This is one reason why tools that allow end-users to produce their own reports (Discoverer, ADI, et. al.) have always made me (and, I'm sure other DBAs out there) somewhat nervous.

The other reason I've always been a little twitchy about those tools is accuracy. With professional developers, they understand the need for testing and accuracy. End-users, however, frequently don't have that same appreciation. So, when you allow end-users to develop their own queries and reports, or to extract and manipulate data in a spreadsheet, what kind of risks are you taking?

CIO Magazine has a thought-provoking article on this. Definitely worth a read.

http://www.cio.com/article/print/699592

– James

Adventures in SQL*Net encryption (E-Business Suite edition)

This is a strange problem that we encountered at one of my clients recently. Basically, they're turning on the SQL*Net encryption features of Oracle Advanced Networking Option in their R12.1.3/11.2.0.3 environments. To do this, we're following note 376700.1 "Enabling SSL in Oracle E-Business Suite Release 12". In particular, Section 10 of that document.

Everything was working well. E-Business Suite was able to connect just fine, even our developers were connecting through TOAD. The only significant problem we encountered was with the Windows-based Workflow Builder 2.6.3.5 client. Here, despite a number of attempts at playing with the sqlnet.ora settings, we were consistantly getting the "ORA-12599 TNS:cryptographic checksum mismatch" error.

The mandate from our internal security team was that all outside traffic had to be encrypted. Since we have an external "DMZ" appsTier node, that meant that we had to turn on this SQL*Net encryption. For ALL of our environments.

Naturally, we opened a SR. As time drug on, we started talking about other options. Do we create one non-production environment entirely behind the firewall for the developers to use that doesn't have the encryption turned on? What about making them ssh to the box directly and use WFLOAD?

Finally, after a couple of months of back-and-forth with the support analysts, plenty of "level 16" traces, and a couple of debates, support was able to supply a solution.

With ANO, you specify params like this in the sqlnet.ora file:

#

# Server Settings

#

SQLNET.ENCRYPTION_TYPES_SERVER=(AES256,AES192,3DES112,3DES168)

SQLNET.ENCRYPTION_SERVER=REQUIRED

SQLNET.CRYPTO_CHECKSUM_SERVER=REQUIRED

SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER=MD5

In the case of the "ENCRYPTION_TYPES_SERVER", this is a list of encryption algorithms that the server will accept. The server will negotiate the strongest encryption algorithm with the client.

On the client, you would have similar parameters in the sqlnet.ora file (well, you don't really need them because in many cases things will auto-negotiate based on defaults):

#

# Client Settings

#

SQLNET.ENCRYPTION_TYPES_CLIENT=(AES256,AES192,3DES112,3DES168)

SQLNET.ENCRYPTION_CLIENT=REQUIRED

SQLNET.CRYPTO_CHECKSUM_CLIENT=REQUIRED

SQLNET.CRYPTO_CHECKSUM_TYPES_CLIENT=MD5

Again, the ENCRYPTION_TYPES_CLIENT, this is a list of the algorithms that the client "can speak". Now, in the case of Workflow Builder 2.6.3.5 (which installs with a 10.1.0.2 client), it appears that the client is negotiating "in bad faith". Basically, it is allowing the negotiation to take place "normally". Based on that, the server will choose the strongest encryption from the ones common to both client and server. Unfortunately, the even though the client *says* it can handle AES256 (and others), it actually can't. So, for the Workflow Builder client, you are limited to the much older RC4_* encryption algorithms.

So, for Workflow Builder 2.6.3.5, here is what we have found to work for the client-side sqlnet.ora:

#

# Client Settings

#

SQLNET.ENCRYPTION_TYPES_CLIENT=rc4_256

SQLNET.ENCRYPTION_CLIENT=REQUIRED

SQLNET.CRYPTO_CHECKSUM_CLIENT=REQUIRED

SQLNET.CRYPTO_CHECKSUM_TYPES_CLIENT=MD5

You also need to make sure that the server-side sqlnet.ora contains the following line (or, at least contains the RC4_* encryption type you're intending to use):

SQLNET.ENCRYPTION_TYPES_SERVER=(AES256,AES192,3DES112,3DES168,RC4_256,RC4_128,RC4_56,RC4_40)

NOTE: In my testing, these algorithms work RC4_40, RC4_56, RC4_128, RC4_256 (we're using RC4_256 because it is the strongest of the RC4_* choices). The various RC_* encryption types are NOT mentioned in the sample sqlnet.ora provided by 376700.1 and, given their age, I would NOT recommend using them in a production environment.



– James

Certification on Oracle (RedHat) Enterprise Linux 6?

One of the questions that getting with increasing frequency is "Can we upgrade our systems to Oracle (or RedHat) Enterprise Linux 6".

As always, my first response is to jump on My Oracle Support, navigate to the "Certifications" tab, and attempt to query up the certified platforms for the Oracle Server (RDBMS). In my experience, the Oracle Server is usually among the first products certified on any (major) platform.

So, I go to the Certify page on MOS, and it appears that Oracle (or RedHat) Enterprise Linux 5 is the most recent certified with the current version of the database (11.2.0.3).

Now, I'm thinking to myself, "Hasn't it (Linux 6) been out for some time?". Turns out, it has. RedHat Enterprise Linux 6 was initially released in November, 2010.

So, a with a little effort ("the Google knows all"), I was able to find a few tidbits:

First, from a RedHat Blog Post back in August, 2011:

We’re pleased to announce that on Tuesday, August 9, we formally submitted to Oracle full certification test results of the Oracle 11gR2 database (Single Instance and RAC (including ASM) for x86 and x86-64) on Red Hat Enterprise Linux 6. Oracle database certification is a self-certification program whereby operating system vendors perform extensive testing and submit the results to Oracle for audit and approval.

The certification process we conducted with Oracle 11gR2 and Red Hat Enterprise Linux 6 is the same process we have successfully completed a number of times with earlier Red Hat Enterprise Linux releases. With those releases, Oracle’s certification approval process took about six weeks from the day we submitted test results to Oracle to the day that Oracle posted the certification on their MetaLink support site (https://support.oracle.com). Based on this experience, we would expect certification of the 11gR2 database on Red Hat Enterprise Linux 6 to occur sometime in CYQ4 of this year. We look forward to Oracle’s response and to working with them to complete this certification process. In the interim, customers may also contact Oracle directly for updates on the certification status at gcp-customerfeedback_us@oracle.com.

In addition to the certification testing described above, we perform ongoing and extensive testing on Oracle 11gR2 at every minor release of Red Hat Enterprise Linux. Consequently, we confidently recommend the deployment of Oracle 11gR2 in Red Hat Enterprise Linux 6 production environments today.

And also, from a note on My Oracle Support:

Database Client or Database Server Install on Red Hat Enterprise Linux 6 (RHEL6) or Oracle Linux 6 [ID 1350000.1]

At the time of the last update of this article (30-Jan-2012), Red Hat Enterprise Linux 6 (RHEL6) and Oracle Linux 6 are not certified or supported for use with any Oracle Database version. Be sure to use only certified/supported combinations of Database version and OS version, which you can find under the Certifications tab of My Oracle Support (MOS). Please note that although Certify used to have a status of "Planned" for Red Hat Enterprise Linux 6 (RHEL6) / Oracle Linux 6, this does not imply any guarantee of certification. Also, please do not open a Service Request asking when the OS will be certified. Any additional information will be added to Certify when it becomes available, and Support is unable to provide any more information than what is there. The Certify information on MOS is the only official source for Oracle certification.

So there you have it. Oracle Server (and, consequently, E-Business Suite) are NOT currently certified to run on either Oracle Enterprise Linux 6.x or on RedHat Enterprise Linux 6.x. Unfortunately, anyone you ask at Oracle will basically tell you that "due to revenue recognition rules, we can't say".

What I can tell you is, despite the "this does not imply any guarantee of certification" message, I think it highly likely that they will certify it (eventually). Especially given Oracle's large Linux installed base (and the fact that Oracle's own Linux distribution has so much in common with RedHat's).

Meanwhile... we wait.

-- James

UPDATE:  I sent a few emails and heard back from some people at Oracle.  From the sound of it, a certification announcement for Oracle Enterprise Linux 6 is being "planned" but, naturally, they can't provide any date information.  But, when you combine this with the information above, it leads me to believe that some certification might be coming sooner rather than later.  Let's keep our fingers crossed!

UPDATE #2:  Oracle has now (22 March, 2012) certified 11.2.0.3 against Oracle Enterprise Linux 6 (Using the Unbreakable Enterprise Kernel V1) for x86-64 platforms. (Press Release).  They have also announced that certification against RHEL 6 will be complete within 90 days.

UPDATE #3 (6/27/2012):  Oracle has just announced certification for Oracle Enterprise Linux 6.0 (x86-32), Red Hat Enterprise Linux 6.0 (x86-32 and x86-64), and  Novell SUSE Linux Enterprise Server (SLES) version 11 (64-bit).  See Steven Chan's blog for more details:  https://blogs.oracle.com/stevenChan/entry/oracle_e_business_suite_release3

Changes are coming to My Oracle Support

Oracle is in the process of getting rid of the incredibly annoying "Flash-based" version of My Oracle Support and replacing it with a site developed using their own Application Development Framework (ADF).

Over the weekend of January 27, 2012, they deployed the new site to replace their "html-based" version. It is available at: https://supporthtml.oracle.com

It's my understanding that, at some future date, the main flash-based site (https://support.oracle.com) will be replaced by this new version.

So, go check it out! It works pretty well... (and, for those of you using "i" devices that can't use Flash, it's a viable alternative...)

I should also note that Oracle does have a mobile version of My Oracle Support. While the mobile version does not offer the full functionality of the normal site, you can at least view and update your SRs. This feature, alone, can be a godsend when you're working on that Severity 1 SR!

The mobile version of My Oracle Support can be found at: https://support.oracle.mobi

– James

Quick Status Update

Just a quick status update. After the dissolution of TriOra Group, I became an independent contractor working with RedRiver Solutions on most projects. RedRiver Solutions is a Dallas, TX based IT Consultancy specializing in Oracle and Oracle E-Business Suite.

– James