A Lifeline for EBS Customers Still on RDBMS 11gR2 and 12cR1

Yesterday, the Oracle E-Business Suite Applications Technology Group made a couple of very significant announcements for E-Business Suite customers.

• RDBMS 12cR1 (12.1.0.2) Extended Support is available through July 31, 2022
• The Extended Support fees have been waived for 12cR1 databases used for E-Business Suite through July 31, 2022.  [Previously, December 2020]

• RDBMS 11gR2 (11.2.0.4) Extended Support is available through December 31, 2020.
• The Extended Support fees have been waived for 11gR2 databases used for E-Busines Suite through December 31, 2020.

The current certified versions of the Oracle RDBMS for use with E-Business Suite R12.1 and R12.2 include 11gR2, 12cR1, and 19c.

What should E-Business Suite customers do now?

If you're on 12cR1, you should upgrade to 19c as soon as reasonably possible.
If you're on 11gR2, you too, should upgrade to 19c as soon as reasonably possible..  You could upgrade to 12.1.0.2, but you would be forcing yourself into another upgrade (to 19c) pretty soon.

Should I wait for the next release?

The Oracle Server group has switched to a "yearly" release schedule.  So, sometime during 2020, there should be an "Oracle RDBMS 20c" and, sometime in 2021, we should expect an "Oracle RDBMS 21c".

However, given that it takes a while to certify the database against E-Business Suite on the variety of platforms that are necessary, the E-Business Suite team has indicated their intention to certify on every other release.  Also important is that the certification for E-Business Suite tends to lag the database release by at least 6 months.  So, that means that the next certified release of the Oracle RDBMS for E-Business Suite should be RDBMS 21c and you might not see that certification until late in 2021.

So, if you choose to wait for the next release, you may find yourself praying for another extension... which is increasingly unlikely.

References

• Release Schedule of Current Database Releases (Doc ID 742060.1)
• Extended Support Fee Waiver for Oracle Database 12.1 and 11.2 for Oracle E-Business Suite (Doc ID 2522948.1)
• Oracle Applications Technology Group Announcement
• Oracle Support Blog Announcement
• Oracle Lifetime Support Policy for Technology Products
• Oracle Lifetime Support Policy for Applications

Windows 10 and E-Business Suite

Tomorrow (July 29, 2015) we will see the release of Windows 10 into the wild.  PC's all over the world that aren't under "enterprise control" have been signing up to download the update through Microsoft's "Windows Update" delivery mechanism and it is "going to drop" on July 29^th, 2015.

For us (E-Business Suite DBAs) there should be at least some level of concern.

Because they are using this mechanism, this will likely be the largest mass deployment of software that anyone has witnessed.  Also, due to the delivery method, I would expect that Windows 10 uptake will significantly surpass any other Windows roll-out in history.

Remember, with previous Windows upgrades, you either had to go out and purchase the software or you got it when you acquired a new laptop.  This meant that, with few exceptions, those of us who have to deal with these changes at least had some breathing room before we really felt it.

Beginning tomorrow, however, anyone with a licensed copy of Windows 7 (SP1), Windows 8, or Windows 8.1, could potentially upgrade to Windows 10.  While this really isn't a concern for most corporate PC's (where software updates/upgrades are managed by the centralized IT department), if you're on a project where users and/or consultants have "unmanaged" PCs, you could encounter some questions.

The first thing you need to know is that, of course, nothing about Windows 10 has been officially "certified" by anyone at Oracle yet.  So, you could always stand behind that statement.  And, certainly, if your IT department is even considering rolling out Windows 10 to anyone, they should wait until that certification information is released.

Now, for those of you who are just wondering, does it even work?  The answer is, yes, it appears to... but there are a few things you should know.

First, Windows 10 ships with a new minimalistic browser called "Microsoft Edge" (also known as "Project Spartan").  The browser works pretty well and the interface is clean, which is nice.  But, Microsoft Edge doesn't support plug-ins (specifically, it no longer supports Active-X).   This means that you will be unable to launch Java from the Edge browser which, in turn, means that you won't be able to launch Oracle Forms from within E-Business Suite.

But, never fear.  Windows 10 also ships with Internet Explorer 11, which is certified in many E-Business Suite configurations.  I performed some rather limited testing (log into R12.1.3, launch forms, basic navigation) using a recent pre-release version of Windows 10 (x86-64, build 10301) and Java JRE 1.8.0_51 (32-bit) and everything appears to function without much issue.  Obviously, this was far from a complete test and I wouldn't go into production with it.  Fortunately, since Oracle has already certified Internet Explorer 11 on Windows 7 and 8.1 (notably, not Windows 8) with EBS, I doubt that certification for IE11+EBS+Windows 10 will find too many problems.

So, the long and short of it is, should you encounter that Windows 10 early adopter, they should have some luck using Internet Explorer 11 (assuming that you're patched up to support it per MOS 389422.1, of course).    Of course, should the user be technically inclined and still want to remain an early adopter, I strongly recommend running an older version of Windows (Windows XP or Windows 7) in a VirtualBox VM.   It's a great way to be current and still be able to use some of the really old tools.  (Workflow Builder, anyone?)

E-Business Suite R12.2 Is now available!

The long-awaited update to E-Business Suite R12.2 is now available. I'm sure we'll hear more at OpenWorld this week, but until then:

The press release can be found here:

http://www.oracle.com/us/corporate/press/2016931

Also, it is available for download on E-Delivery:

Oracle Database 12c Launch Webcast

In my post yesterday, I indicated that Oracle Database 12c (12.1.0.1.0) was available for download on Oracle's E-delivery and TechNet websites, but that it still hadn't officially been "launched".  The official "product launch" take place on July 10, 2013.  You can RSVP for the launch webcast through the link below:

Launch Webcast:  http://tinyurl.com/ow28hr2

Yesterday's Blog post:  http://thedbalife.blogspot.com/2013/06/oracle-database-12c-is-available-for.html

-- James

Oracle Database 12c Is Available for Download

File this under "it's about time" and "ICYMI (In Case You Missed It), but Oracle has released Database 12c (12.1.0.1.0). Downloads can be found on their TechNet and E-Delivery sites. At this point, the only available versions are for Linux (x86-64), Solaris (Sparc64), and Solaris (x86-64). Other platforms will surely follow.

Not officially released... yet

According to media reports (and my inability to find an actual press release from Oracle), the formal launch of Database 12c will occur "within a couple of weeks".

Differences between TechNet and E-Delivery

While, otsensibly, it may be the same software, there is always the possibility that you'll get slightly different versions. The software that you download from TechNet is usually in the form of either a zip file or a "tarball" of the staged installation. The downloads from E-Delivery are also zip files, but they represent the actual media packs (CD or DVD). For some reason, Oracle doesn't do ISOs, but, nevertheless, the E-Delivery downloads are typically viewed as more "supported". As a result, I recommend using the E-Delivery downloads rather than TechNet if you're planning on doing anything that is going to need to be handled under a support contract.

Naturally, for either method, you will have to agree to license terms and export conditions. If you have never used E-Delivery from your Oracle account, there might be a slight delay as your account is verified by Oracle.

As with all new software, be sure to test thoroughly and make sure any applications are certified with 12c before deploying to production.

Oracle Client 12c is also available

The Oracle 12c Client can also be downloaded for the following platforms: Linux (x86-32), Linux (x86-64), Microsoft Windows (x86-32), Microsoft Windows (x86-64), Solaris (Sparc 64), Solaris (Sparc 32), Solaris (x86-32), Solaris (x86-64).

NOT CERTIFIED WITH E-BUSINESS SUITE

Since this blog is focused on E-Business Suite (and E-Business Suite is what I do), I feel the need to state that Database 12c is NOT certified with ANY RELEASE of E-Business Suite at this point. I suspect that we'll see it certified against 12.1.3 and the upcoming 12.2 at some point in the future (maybe 12.2 on release). It is highly unlikely (in my opinion) to be certified against any release 11i. In the event that it is certified against 11i, you can bet that it will be a pretty low priority item.

You can find them available here:

Oracle E-Delivery: https://edelivery.oracle.com

Oracle TechNet: http://www.oracle.com/technetwork/indexes/downloads/index.html

-- James

Deciphering support and licensing issues surrounding Oracle on VMWare

I frequently run into clients that are wanting to run Oracle products in their VMWare cluster. Since I generally deal with E-Business Suite customers, I tend to take the position of "anything that swallows machines whole should probably have a physical machine" approach to production systems. However, I can see some distinct advantages to virtualization, particularly when it comes to managing numbers of non-production environments.

Unfortunately, there is a lot of confusion out there as it relates to Oracle and virtualization... particularly surrounding one of the most popular virtualization solutions, VMWare. I'll try to provide my best understanding of the issues here.

Are Oracle products certified on VMWare?

The short answer is, NO. But, that really shouldn't be that much of a concern. Keep in mind that a VMWare Virtual Machine is, technically, hardware. Oracle doesn't tend to certify against hardware. And that's what that VMWare really is, it's "virtual hardware". As such, it's really no different than a particular model of Dell or HP ProLiant.

What Oracle does do is certify against a platform. A platform is the combination of a particular version of an operating system (Solaris 10 vs. Solaris 11, for example) and processor architecture (Sun SPARC vs. Intel x86-32 or Intel x86-64). In the case of a deployment to VMWare, your platform will be determined by the operating system that you intend to run inside of the virtual machine. (For example, Red Hat Enterprise Linux 4/5/6 for x86 or x86-64).

Are Oracle products supported on VMWare?

Oracle's official support position can be found in MOS Note 249212.1, copied below (emphasis mine):

Support Position for Oracle Products Running on VMWare Virtualized Environments [ID 249212.1]

Purpose

---------

Explain to customers how Oracle supports our products when running on VMware

Scope & Application

----------------------

For Customers running Oracle products on VMware virtualized environments. No limitation on use or distribution.

Support Status for VMware Virtualized Environments

--------------------------------------------------

Oracle has not certified any of its products on VMware virtualized environments. Oracle Support will assist customers running Oracle products on VMware in the following manner: Oracle will only provide support for issues that either are known to occur on the native OS, or can be demonstrated not to be as a result of running on VMware.

If a problem is a known Oracle issue, Oracle support will recommend the appropriate solution on the native OS. If that solution does not work in the VMware virtualized environment, the customer will be referred to VMwar for support. When the customer can demonstrate that the Oracle solution does not work when running on the native OS, Oracle will resume support, including logging a bug with Oracle Development for investigation if required.

If the problem is determined not to be a known Oracle issue, we will refer the customer to VMware for support. When the customer can demonstrate that the issue occurs when running on the native OS, Oracle will resume support, including logging a bug with Oracle Development for investigation if required.

NOTE: Oracle has not certified any of its products on VMware. For Oracle RAC, Oracle will only accept Service Requests as described in this note on Oracle RAC 11.2.0.2 and later releases.

In my understanding of the actual way that the policy is applied, it's really a matter of whether or not the support engineer suspects VMWare to be the culprit. What I'm saying here is that, generally speaking, the support engineer will work your issue the same way that he/she would if you were on physical hardware. However, once that engineer thinks that VMWare could be the cause of your problem, they reserve the right to "punt" and say "call us back once you've reproduced it on physical hardware".

Now, VMWare, to their credit, has a policy that they call "Total Ownership", where they will accept accountability for any Oracle-related issues. You can read their official policy at the link below.

http://www.vmware.com/support/policies/oracle-support.html

It is my understanding that, as part of the "Total Ownership" policy, VMware will reproduce the problem on physical hardware for the customer if Oracle decides that VMWare is the problem.

What about Licensing?

Part of the big problem I've always had with Oracle on VMWare is caused by Oracle's per-CPU licensing policy. My original understanding was that, if you have a total of 64 cores in your VMWare cluster, it didn't matter if you were only using 8 cores for Oracle. Oracle would tell you that you had to pay for 64 cores. The idea behind this is that you could, potentially, resize the virtual machine to suit certain needs. Maybe you need more horsepower during month end?

What I've since learned is that Oracle has a policy document (below) that talks about "soft" vs. "hard" partitioning.

http://www.oracle.com/us/corporate/pricing/partitioning-070609.pdf

What I've described above would fall under the concept of "soft partitioning". However, "hard partitioning" methodologies allow for a different approach. VMWare has (naturally) a nice document that explains their approach to implementing clusters that are in compliance with Oracle's licensing requirements.

http://www.vmware.com/files/pdf/techpaper/vmw-understanding-oracle-certification-supportlicensing-environments.pdf

From that document, pay particular attention to section 2.2. In that section (specifically Scenario B), they discuss DRS Host Affinity rules and VMWare CPU pinning. (emphasis mine)

2.2 Clusters: Fully Licensed Versus Partially Licensed Clusters

Scenario B: Partially Licensed Clusters

When a customer does not have enough Oracle application instances to justify creating a dedicated cluster for those applications, only a subset of the hosts in the cluster are licensed for the Oracle application. In this situation, the customer must be careful to restrict the movement of Oracle application instances and virtual machines to only those hosts that are licensed to run the product.

In this case, DRS Host Affinity rules can be used to appropriately restrict the movement of virtual machines within the cluster. DRS Host Affinity is a vSphere feature that enables you to ensure that your Oracle applications are restricted to move only between a subset of the hosts—that is, not all hardware in the cluster is “available” to the Oracle software. DRS Host Affinity is a clustering technology and is not a mechanism for soft or hard partitioning of the servers. As explained in section 2.1, using VMware CPU pinning to partially license a host is not currently recognized by Oracle as a “hard partitioning” mechanism that receives subsystem pricing. However, once you have fully licensed the host, you have the right to design your environment such that the Oracle workloads are free to run on the licensed hosts inside the cluster. At present, Oracle does not have any stated policy regarding clustering mechanisms or DRS Host Affinity. Customers can easily maiatain records for compliance purposes as explained in section 2.3.

The advantages of this approach are similar to the advantages achieved with a fully licensed cluster. Because customers are typically able to increase the utilization of licensed processors, they reduce license requirements. However, consolidation ratios tend to be lower, because advanced vSphere features can be employed only on a smaller subset of the hosts.

VMWare CPU pinning is a feature that (in my understanding) would allow you to say that a given VM would only use certain cores in a physical host. So, if you have a single host with 16 cores, you can "pin" a given VM to four of them. According to Oracle's partitioning document (and VMWare's document), you would still be required to pay for all 16 cores in the box. The basic logic here is that Oracle's licensing policy is based on the number of cores in a physical server. You can't license part of a box. Period. No exceptions.

On the other hand, DRS Host Affinity, is a way to pin a virtual machine to a given host (or collection of hosts) within a cluster. So, let's say that you have ten (10) 8-core physical hosts (total of 80 cores) in your VMWare cluster. Using DRS Host Affinity, youcould restrict your Oracle VMs to a subset of those physical hosts. For example, if you restricted your Oracle VMs to only five (5) of those physical hosts, VMWare's contention is that you would only have to license 40 cores.

I sould probably include the standard "IANAL" (I am not a lawyer) disclaimer. I'm also not a VMWare administrator. What I am is a DBA and an IT Geek. That's pretty much the limit of it.

Hopefully this provides some clarity on the issue.

For further reading on the subject, here are a couple of blog links that I used in my research:

http://blogs.vmware.com/apps/2012/11/update-on-virtualizing-oracle.html

http://longwhiteclouds.com/2012/07/21/fight-the-fud-oracle-licensing-and-support-on-vmware-vsphere/

– James

Why I don't depend on TOAD (or OEM) and neither should you.

My apologies in advance, as this posting may sound like something of a rant.

The first thing I'd like to point out is that I have no real problem with TOAD, Oracle Enterprise Manager, or Windows-based editors. They are all excellent tools that can be extremely helpful in your environment. My objection to these tools is based solely on a lowest-common-denominator argument.

First, a little background. Back in the early 1990's, I was working as a Unix Systems Administrator for a company in Kansas City, MO. Since then, I've worked mainly as a consultant.

Shortly before I started that job in Kansas City, the company had hired a new CIO who let go about half of the legacy (mainframe, COBOL) IT department. The new direction for the company was implementation of Oracle E-Business Suite on Data General Unix (DG/UX).

The mainframe IT staff that survived were being re-trained in the new technology. At one point, several of them came to me insisting that I install ISPF (an editor they were used to on the mainframe) onto the DG/UX boxes because they were struggling to learn to use the vi editor. I informed them that, while they (as a group) may carry enough weight to convince the CIO to direct me to install it (assuming it was even available). However, when they go to their next job and claim that "they know Unix", they would be alone and wouldn't have that leverage.  My suggestion was that I would help them to learn the vi editor. (I did offer emacs as an alternative, since it is and was extremely common on Unix systems... Unfortunately, friendlier editors like pico, nano, and joe didn't exist yet.)

If your primary job is software development, a tool like TOAD is generally something you can depend on having. However, as a DBA, you can't necessarily depend on having TOAD (or even Oracle Enterprise Manager) at your disposal at all times. Maybe you're starting a new job and the previous DBA hadn't set up Enterprise Manager (or you haven't gotten around to it yet). Even in environments where those tools are available, they may or may not be working when you need them.

So, my advice? There are certain tools that are almost ALWAYS there. Get comfortable with ssh, SQL*Plus, and vi (or vim).  They are your friends.

– James

Oracle Support Changes for EBS (part deux)

On Monday, Cliff Godwin announced some significant changes to E-Business Suite support. The official announcement is now available on MOS (Note: 1495337.1). The text of the announcement is below:

As part of Oracle’s continued commitment to our customers, we will be providing an exception for the first 13 months of Sustaining Support on Oracle E-Business Suite Release 11.5.10 (11i10), valid from December 1, 2013 – December 31, 2014. This exception support will be comprised of three components: (1) new fixes for Severity 1 production issues, (2) United States Form 1099 2013 year-end updates, and (3) payroll regulatory updates for the United States, Canada, United Kingdom, and Australia for fiscal years ending in 2014.

In addition, the Extended Support period for E-Business Suite Release 12.1 has been extended through December, 2018. Customers with an active Oracle Premier Support for Software contract will automatically be entitled to Extended Support deliverables for E-Business Suite 12.1.

NOTE: The changes to 11i Sustaining Support do NOT extend to the security patches. Oracle will NOT provide Quarterly CPU patch for 11i will come out in October, 2013. The one exception to this is that they will provide fixes for "P1" security issues.

– James

E-Business Suite Current Release and Roadmap

Current release is R12.1.3 (but you already knew that!). Enhancements coming with R12.2 include Online Patching (20 years in the making) based on a database feature specifically requested by the E-Business Suite group called "Edition-Based Redefinition". The expectation is that you will be able to apply patches of any size with minimal downtime (less than an hour, target is more like 15 minutes).  (I will discuss this feature in a more in-depth posting later).

Other enhancements include integration with Endeca to provide an e-commerce shopping experience to many of your ERP business processes. (I've heard the "shopping for shoes" reference three times already at OpenWorld when discussing this feature). Cliff Godwin provided an interesting demo of this feature during his presentation earlier today. It is expected that this capability will be released "within the year". Cliff made a point to indicate that initial releases will cover several modules (I believe it was eight at initial release) but that new modules and Endeca-related features should come along quickly (due to the rapid development cycles available with Endeca). He was unclear, however, as to whether the Endeca feature set will be available on releases prior to R12.2.

– James

Oracle Support Changes Announced for EBS

During his session earlier today, Cliff Godwin announced some significant changes to the E-Business Suite support deadlines. I will put together a more polished post later, but for now, here are the details:

EBS 11i (11.5.10.2) is STILL on Sustaining Support until December 31, 2013. However, they are making a few exceptions to the Sustaining Support policy. First, they will continue to provide bugfixes for Severity 1 production problems until December 31, 2014. Secondly, they will continue to provide updates for US Form 1099 through 2013 Year End. And finally, they are committing to provide Fiscal Year End 2014 Payroll updates for US, Canada, and UK.

For customers on Release 12.1, Oracle is waiving the Extended Support fees until December of 2018.

-- James

MINOR CORRECTION: EBS 11i is not "still on sustaining support". EBS 11i is actually on Extended Support until Nov 2013. (Thanks, Srini, for pointing this out!)

Impact of EBS R12.2 changes – WebLogic

A number of very smart people seem to think that the introduction of WebLogic is going to be a major "game changer" for E-Business Suite DBAs. There is certainly some truth to that thought. WebLogic is a completely different animal than the Apache or Oracle Application Server that we've gotten accustomed to in E-Business Suite 11i and R12.

But consider this. I suspect that most environments are fairly vanilla. When it comes to the basic degree of interaction that an E-Business Suite DBA has with the Applications Server, I suspect that the change will be largely transparent. Consider what you do with the current Applications Server product. Start and stop? That's done through AutoConfig managed scripts. Basic configuration for EBS? That's handled through AutoConfig. Why would we expect these things to change just because they've changed out the tool?

What will it change?

I'm certain that the log files will be different. The naming convention will probably change and the location of the files might change. The content of the files will certainly be different. As will the configuration files (but, again, I would expect those to be AutoConfig managed).

The big place where you will notice the WebLogic change (from an Administrator standpoint) is if you're trying to do something else with it. If you're dealing with Discoverer or deploying APEX, for example, you'll certainly get up-close-and-personal with WebLogic.

So, should you be afraid of the WebLogic change? I don't think so. Yes, custom deployments (anything that actually requires you to "deep-dive" into the configuration of WebLogic) will have a learning curve. That said, many of the components of Oracle Fusion Middleware already use WebLogic and, if you're doing any custom deployments, odds are you're also already dealing with WebLogic elsewhere and will have some familiarity by the time you go to tie it into E-Business Suite.

When you factor in these things with the fact that WebLogic is a more mature and overall better product than the current applications server used in E-Business Suite, I think that the change is probably a good thing.

– James

Speculation about EBS Release 12.2

Floyd Teeter raises some good points in his blog posting out today. You can read the full posting here (http://orclville.blogspot.com/2012/08/i-wonder-what-tomorrow-has-in-mind-for.html). Primarily, that we won't actually have EBS R12.2 released during OpenWorld (as many of us were hoping).

Don't get me wrong, there will certainly be some good EBS R12.2 information out there. Oracle put together a very informative presentation and discussion surrounding the patching changes at Collaborate in Las Vegas this past spring. You can almost certainly bet that there will be an updated version at OpenWorld.

But, considering that the closed Beta program is still underway and the Early Adopter program hasn't started, any path of deduction should tell you that you're not going to have a R12.2 "GA" announcement for a few months at best. Which most means that sometime around Collaborate 2013 (Denver) would be a decent guess.

– James

URGENT BULLETIN: Disable JRE Auto-Update for All E-Business Suite End-Users

This notice just came out on Steven Chan's blog. If you're not following it, you definitely should.

The issue at hand is not that Auto-Update in itself causes problems. The real issue is that Oracle is going to push (through Auto-Update) the JRE 1.7 update. The JRE 1.7 update is NOT certified with E-Business Suite (any versions) at this time.

So, in order to keep your user's desktops on JRE 1.6, you MUST turn this auto-update feature off!

The update for JRE 1.7 could be pushed as early as July 3, 2012. The update WILL definitely be pushed after September 7, 2012 (after the release of JRE 1.6.0_35).

Steven Chan's blog has more information (including instructions on what you can do to undo the JRE 1.7 update if you get hit by it). The full link to the posting on Steven Chan's blog can be found here:

https://blogs.oracle.com/stevenChan/entry/bulletin_disable_jre_auto_update

– James

OAUG Connection Point 2012 in Austin, TX (July 11-12, 2012)

OAUG Connection Point 2012 is coming to Austin, TX, July 11-12, 2012! The conference will be held at the Omni Austin Hotel in beautiful downtown Austin, TX.

The keynote will be given by Jeanne Lowell, Vice President, Oracle E-Business Suite Strategy, Oracle Corporation. (Hopefully we can extract some R12.2 news from her!)

Other featured speakers include:

Elke Phelps, Senior Principal Product Manager, Oracle E-Business Suite Applications Technology, Oracle Corporation. (Elke always gives interesting and informative presentations!)

David Bowin III, Oracle Fusion Applications Product Development, Oracle Corporation

Amrita Mehok, Senior Director, Product Strategy, Oracle Corporation

Some other names you may recgonize will be presenting as well:

Alyssa Johnson, ROLTA (Sessions # 10913 and 10933)

Anne Carlson, Oracle (Session # 10868)

Art Dowd, O2Works (Session # 10829)

Barbara Matthews (Session # 10925)

Bill Dunham, OATC, Inc. (Session # 10915)

John Stouffer (Session # 10880 and 10924)

Michael Barone, OATC, Inc (Session # 10905 and 10907)

Mike Swing, TruTek (Session # 10878)

Susan Behn, Infosemantics, Inc. (Session # 10821)

...and many more!

Oh, and I'll be there too! I'm giving my "Anatomy of an Upgrade to 12.1.3 (including Platform Migration)" presentation (Session # 10802). I'll also be participating in John Stouffer's "R12.1 Upgrade Panel" (Session # 10924) along with Bill Dunham, Mike Swing, and Alyssa Johnson. The panel discussions are always entertaining and provide an excellent way for the community to discuss their challenges and experiences.

"Early Bird" Registration is open through June 24^th. Particularly if you're in or near the Austin, TX area, this can be a very cost-effective way to network with the community and find solutions to your problems. More information on the conference is available through the OAUG website or this direct link:

http://connectionpoint.oaug.org/2012/austin

I hope to see you there!

– James

On DBAs, Developers, and Performance

Cary Millsap has an excellent (as usual) blog posting today about the software development process employed by so many oranizations.

You can read his full posting here:

http://carymillsap.blogspot.com/2012/06/organizational-constraint-that.html

This plays into one thing that I see quite a bit as a DBA at client sites. Most developers that I encounter at client sites don't tend to focus so much on performance (unless it is painful during their testing). This isn't, specifically, their fault. In many cases, the developers are under significant pressure to "make it work" and move onto the next thing. As a result, they don't really have the time to worry about performance. Besides, that's the DBA's job, isn't it?

Well, actually, it isn't. See, here's the thing, from a DBA perspective, we have a relatively small handful of tools at our disposal. From the bottom-up, here is a basic list of things that a DBA generally does to impact the performance of a system:

Properly configure disk and I/O. This is (or should be) really a collaboration between the DBA, the Systems Administrators, and the Storage Administrators. Done correctly, this shouldn't really be a problem. However, as with everything, it is still possible to screw it up. Make sure that you have redundant I/O paths that have sufficient bandwidth for your system. Spread your I/O across many physical drives. With RAID technologies (particularly RAID 1/0) this is VERY easy to accomplish. Forget about the old concept of the DBA moving datafiles around to avoid a "hot disk". The disk array can do this at a much more granular and consistent level than any human possibly can. Your primary goal here should be to have as many spindles involved in every I/O operation as possible.

Properly size your hardware. Another collaboration between the DBA and Systems Administrator. Make sure you have adequate horsepower and RAM. And ALWAYS plan for growth! My general recommendation is always to put more RAM into the box than you think you'll need. Given that so many systems that I encounter these days are x86-based Linux systems (rather than "big iron" like Solaris or AIX), memory for these systems is a relatively small portion of their cost. Also, RAM doesn't impact your Oracle licensing!

Properly tune your OS Kernel and Database parameters. I think that this is one area where developers, managers, and users tend to have gross misconceptions. While it's true that tuning a system to truly "optimal" performance is a dark art, the truth is that, unless you've really screwed something up (sized your SGA too small, too few buffers, etc.), odds are you're not going to see huge performance gains by tweaking these parameters. In most cases, "decent performance" is fairly easy to achieve. Now, squeezing out that extra 20%? That can require in-depth analysis of statistics, utilization, I/O patterns, etc. This is where the "dark art" comes into play. And, honestly, this requires time to observe and adjust.

Unfortunately, too many developers, managers, and even users, seem to wonder why that idiot DBA didn't just set the magic "MAKE_SQL_RUN_FASTER=TRUE" parameter. (Shh! Don't tell anyone, that's OUR little secret!)

The truth is, unless something is wrong at these lower levels, the biggest performance gains are going to come from tuning the code. And, in my opinion, since it's the developer's job to produce a quality product, it's ultimately the developer's job to do tune their code.  Unfortunately, as Cary points out, too many organizations are structured in a manner that breaks the feedback loop required for developers to do this properly.

That MUST be fixed.

– James

EBS R12.2 Online Patching Webcast

For those of you who are (like me) anxiously awaiting the release of R12.2, the Applications Technology Group (ATG) is hosting a live webcast to preview the Online Patching feature in R12.2.

Kevin Hudson, one of the lead architects of this particular feature, is the presenter for this webcast.  I had the pleasure of attending Kevin's session at Collaborate 2012 where he discussed this feature in-depth.  He is an excellent presenter and is very well-versed on this topic.  I'm sure this webcast will be worth your time.

This two hour webcast will take place on June 14 @ 8:00am (Pacific Standard Time).

Full details are on Steven Chan's blog at the link below.

ATG Live Webcast June 14: Technical Preview of EBS 12.2 Online Patching

-- James

E-Business Suite and the 32-bit vs. 64-bit question

Before I get flamed on this, I want to make clear that, for the purpose of this posting, I'm speaking specifically about operating systems (not hardware). Most of the hardware being sold today is already 64-bit, however, you can run most 32-bit operating systems on 64-bit hardware. It's that distinction that I'm discussing here.

The first thing that you need to know here is that the big benefit of using a 64-bit operating system really is memory. In particular, it is not about the total amount of memory that can be installed in the machine (that tends to be hardware), but, about the addressable size of "per process" memory.

In the case of components such as those used on an appsTier in EBS, per-process addressable memory doesn't matter so much, as each process has it's own private memory (and isn't depending on "shared memory" like the database server is). So, aside from the fact that it makes our life much easier from an administrative standpoint (and the industry is going that way), there really isn't much of technical advantage to a 64-bit appsTier.

For EBS 11i (where the DB is certified on x86-64, but the appsTier is only certified on x86-32), you can still use much more than 4GB on an appsTier node (the operating system has a way of addressing large memory). It's just that the amount of memory that can be addressed by a single process is limited to something between 3 and 4 GB.

In the case of EBS R12, the appsTier binaries are still 32-bit, even when you're running on a 64-bit operating system. This makes sense because the only component that can really take advantage of it is the database (because the database processes all attach to the same large chunk of memory [the SGA]).

Note that EBS R12 appsTier is certified on both Linux x86-32 and Linux x86-64.

So, for 11i, the best that they can hope for is to have a separate dbTier (database only) running on Linux x86-64 and use Linux x86-32 for their appsTier nodes. Remember, that the 11i appsTier is NOT certified on Linux x86-64. That doesn't mean that it can't be done, but I seriously doubt that Oracle has any intention to certify a release that old on, what is effectively, a different platform. In both cases, they can/should be 5.X (5.7 is current). Having, effectively, two different platforms will be something of a headache from a Linux administration standpoint, but it's something that they'll have to deal with.

When they get to R12, they should use Linux x86-64 on all tiers (to simplify administrative tasks, as well as being "among the mainstream" of installations). Keep in mind that 64-bit is where "the market" is going. Even though you can (It is certified) do R12 on x86-32, you're better positioned if you're on x86-64.

– James

Vegas, Baby!

On Saturday night, I head to Las Vegas for Collaborate 2012 (http://collaborate.oaug.org). For the Oracle E-Business Suite community, this is one of the biggest events of the year.

For the uninitiated, Collaborate is the annual conference hosted by the Oracle Applications User's Group (http://www.oaug.org). Unlike Oracle OpenWorld, which is largely an Oracle Corporation marketing event, Collaborate is a community-oriented conference. While there are Oracle Corporation presentations focusing on the latest and greatest products and features, the vast majority of the papers and presentations at Collaborate are from actual users of the Oracle products.

Many of these are actual war stories. Walk-throughs of actual implementations and upgrades. Solutions that that others in the E-Business Suite community have put together for their sites to solve real-world problems that they've encountered. Very often, these are things that are not covered (or are covered poorly) in Oracle-provided documentation.

And it's not just E-Business Suite DBA topics, either! There are plenty of presentations on module-specific issues given by others who have already "been there, done that".

In addition to the standard white-paper presentations, there are also panel discussions and Special Interest Groups (SIGs). The panel discussions and SIGs are usually very lively Q&A sessions with recognized industry experts on a wide range of topics.

Planning an upgrade? There's the two-part OAUG Upgrade SIG (Sessions #10522 and #10722)

Having troubles with cloning E-Business Suite? There's an E-Business Suite Cloning Panel. (Session #9360)

Thinking of moving to Linux? There's a panel for that too! (Session #9361)

I will be giving my presentation, "#9554 - Anatomy of an Upgrade to 12.1.3 (Including Platform Migration)" on Wednesday, April 25^th from 11:00am to 12:00pm i(South Seas E) and I will also be on the Migrations to Linux Panel (#9361), also Wednesday, from 1:00pm to 2:00pm (South Seas F).

And, while you're at it, stop by the RedRiver Solutions booth (Booth #1226). I will be hanging out there and helping to pass out updated versions of John Stouffer's books Oracle E-Business Suite Patching 101 and Oracle E-Business Suite Apps DBA 101.

I hope to see you there!

– James

E-Business Suite R12.1.1 is Certified on Oracle Linux 6!

Back in February, I blogged about the pending certification of Oracle E-Business Suite on Oracle Enterprise Linux 6 and RedHat Enterprise Linux 6. In that blog post, I noted that the certification announcement was "planned" but, of course, Oracle doesn't provide dates.

Well, guess what? The waiting is finally over. As these things go, the announcements come out in parts.

First, on March 22, 2012, Oracle announced that Oracle Database 11gR2 and Fusion Middleware 11gR1 were certified. (The press release can be found here.)

And today (April 4, 2012), through the Oracle E-Business Suite Technology blog (known to many of us as "Steven Chan's blog"), we have the E-Business Suite announcement (available here)!

While this is fantastic news, read the announcements carefully!

These certifications are ONLY for Oracle Enterprise Linux 6 on the x86-64 with the Unbreakable Enterprise Kernel (UEK) version 1.

This means that if you're on the x86-32 platform, or if you're on RedHat Enterprise Linux 6, you'll have to wait a bit longer. The sames is true for customers on Oracle Enterprise Linux 6 who have NOT upgraded to the Unbreakable Enterprise Kernel (UEK) version 1 at ALL, or have already upgraded to the Unbreakable Enterprise Kernel (UEK) version 2 which was released on March 13, 2012 (press release here).

According to the database announcement, certification on RedHat Enterprise Linux 6 (and Oracle Enterprise Linux 6 [without UEK]) should be available within 90 days. I would expect the E-Business Suite R12 announcement to follow shortly behind.

What about other E-Business Suite releases? At this point, I have no actual information. But, I can speculate (with a good degree of certainty) that you won't see any 11.5.10.2 certifications against OEL/RHEL 6. E-Business Suite 11.5.10.2 is currently in Extended Support. Even though the support fees have been waived (through the end of Extended Support, November 30, 2012), Oracle doesn't typically certify new platforms once a product goes into Extended Support. (A more detailed discussion of Oracle's recent support announcements can be found here.)

The other question mark out there is OEL/RHEL 6.0 on x86-32. Personally, if you're implementing R12 or upgrading to R12 on Linux, you should be using an x86-64 distribution on x86-64 hardware. However, certification on x86-32 is also forthcoming.

As always, be sure to read/follow the relevant notes through the Certify Tab on My Oracle Support before you start any project to make sure that the combination of components you intend to use are, in fact, certified. These certifications will also detail the various always steps, operating system parameters, packages, and even patches specific to your combination that you will need to follow.

All of this is excellent news, as the OEL and RHEL 5.x line is getting pretty long in the tooth and is approaching it's end of life.

Now... when will we get that R12.2 announcement? Collaborate, maybe? OpenWorld? … the waiting continues.

– James

UPDATE 6/27/2012:  Oracle has just announced certification for Oracle Enterprise Linux 6.0 (x86-32), Red Hat Enterprise Linux 6.0 (x86-32 and x86-64), and  Novell SUSE Linux Enterprise Server (SLES) version 11 (64-bit).  See Steven Chan's blog for more details:  https://blogs.oracle.com/stevenChan/entry/oracle_e_business_suite_release3

Stupid Unix Tricks... Part Two (Remote Command Execution using SSH)

So, let's say that you wanted to have a script on your dbTier that will reach out to your appsTier and shut down the applications. Maybe this is your system-level shutdown script so that when the Unix administrator shuts down the dbTier, everything is shut down nice and neat like...

For the purpose of this exercise, we're going to need to assume that the APPS password is known to the script (how you do that might be the subject of another blog posting). We're also going to assume that the Unix environment is set automatically (and without prompting) on the remote system.

So, how do you do it?

Well, first you have to set up ssh pre-shared keys. This will allow you to login without being asked for a password. (See my earlier posting: Password-less Login Using SSH Pre-Shared Keys)

Once that is configured, you can use a command like this:

ssh applmgr@myappstier.mydomain "cd ${ADMIN_SCRIPTS_HOME};./adstpall.sh apps/${APPSPW}" 2>&1 |tee -a ${LOG}

A few things here. First, you'll notice that I'm actually executing TWO commands remotely. The "cd" to change directories and then the adstpall.sh script (the semicolon allows me to do that in Unix). Secondly, there are environment variables. Here's the thing about those environment variables. In the command above, they are NOT evaluated on the target system. They are evaluated locally on the SOURCE system. If you want to use variables that are local to the target, you're going to have to "escape" them.

For example, this one will use a variable evaluated on the source machine:

ssh applmgr@myappstier.mydomain "echo ${CONTEXT_NAME}"

And this one will use a variable evaluated on the target machine:

ssh applmgr@myappstier.mydomain 'echo ${CONTEXT_NAME}'

Similarly, you can evaluate a variable on the target by “escaping” it:

ssh applmgr@myappstier.mydomain "echo \${CONTEXT_NAME}"

At one client, their standard is to use a script that wraps around the standard "oraenv" to set their environment variables. As a result, every time they log in, they are greeted with a prompt asking them to choose their environment.

This raised an interesting problem for some of the automated processes we were trying to deploy. The automation was driven from a remote box and would need to ssh over to a target box and issue commands. So, how do we configure the environment so that a user logging in interactively is prompted and one issuing a command remotely through ssh isn't? Well, it turns out that, on Linux at least, that remote command doesn't get assigned a TTY. So, we've made a change to the .bash_profile on the target node that looks something like this:

if tty | fgrep pts ; then

#

# Normal, interactive logins

#

export ORAENV_ASK=YES

else

#

# Human-less logins (ssh "command")

# (Suppress output and bypass prompting for oracle environment)

#

export ORAENV_ASK=NO

fi

Now, let's assume you want to be a little more elaborate. You want to clean up extraneous output and capture the results of the command in your logfile (represented by the environment variable ${LOG}):

ssh applmgr@myappstier.mydomain ". ./.bash_profile 2>&1 1>/dev/null;cd ${ADMIN_SCRIPTS_HOME};./adstpall.sh apps/${APPSPW}" 2>&1 |tee -a ${LOG}

Or, maybe you'd like to do something in SQL*Plus on a remote system?

ssh applmgr@myappstier.mydomain “. ./.bash_profile 2>&1 1>/dev/null;sqlplus apps/${APPSPW}” <&1 1>>${LOG}

select sysdate from dual;

EOF

This will redirect stderr to stdout, and send both to your logfile (${LOG}). Pay close attention to the line containing the EOF. It has to be the ONLY thing on the line (not even a trailing space!)

– James