Deciphering support and licensing issues surrounding Oracle on VMWare

I frequently run into clients that are wanting to run Oracle products in their VMWare cluster. Since I generally deal with E-Business Suite customers, I tend to take the position of "anything that swallows machines whole should probably have a physical machine" approach to production systems. However, I can see some distinct advantages to virtualization, particularly when it comes to managing numbers of non-production environments.

Unfortunately, there is a lot of confusion out there as it relates to Oracle and virtualization... particularly surrounding one of the most popular virtualization solutions, VMWare. I'll try to provide my best understanding of the issues here.

Are Oracle products certified on VMWare?

The short answer is, NO. But, that really shouldn't be that much of a concern. Keep in mind that a VMWare Virtual Machine is, technically, hardware. Oracle doesn't tend to certify against hardware. And that's what that VMWare really is, it's "virtual hardware". As such, it's really no different than a particular model of Dell or HP ProLiant.

What Oracle does do is certify against a platform. A platform is the combination of a particular version of an operating system (Solaris 10 vs. Solaris 11, for example) and processor architecture (Sun SPARC vs. Intel x86-32 or Intel x86-64). In the case of a deployment to VMWare, your platform will be determined by the operating system that you intend to run inside of the virtual machine. (For example, Red Hat Enterprise Linux 4/5/6 for x86 or x86-64).

Are Oracle products supported on VMWare?

Oracle's official support position can be found in MOS Note 249212.1, copied below (emphasis mine):

Support Position for Oracle Products Running on VMWare Virtualized Environments [ID 249212.1]

Purpose

---------

Explain to customers how Oracle supports our products when running on VMware

Scope & Application

----------------------

For Customers running Oracle products on VMware virtualized environments. No limitation on use or distribution.

Support Status for VMware Virtualized Environments

--------------------------------------------------

Oracle has not certified any of its products on VMware virtualized environments. Oracle Support will assist customers running Oracle products on VMware in the following manner: Oracle will only provide support for issues that either are known to occur on the native OS, or can be demonstrated not to be as a result of running on VMware.

If a problem is a known Oracle issue, Oracle support will recommend the appropriate solution on the native OS. If that solution does not work in the VMware virtualized environment, the customer will be referred to VMwar for support. When the customer can demonstrate that the Oracle solution does not work when running on the native OS, Oracle will resume support, including logging a bug with Oracle Development for investigation if required.

If the problem is determined not to be a known Oracle issue, we will refer the customer to VMware for support. When the customer can demonstrate that the issue occurs when running on the native OS, Oracle will resume support, including logging a bug with Oracle Development for investigation if required.

NOTE: Oracle has not certified any of its products on VMware. For Oracle RAC, Oracle will only accept Service Requests as described in this note on Oracle RAC 11.2.0.2 and later releases.

In my understanding of the actual way that the policy is applied, it's really a matter of whether or not the support engineer suspects VMWare to be the culprit. What I'm saying here is that, generally speaking, the support engineer will work your issue the same way that he/she would if you were on physical hardware. However, once that engineer thinks that VMWare could be the cause of your problem, they reserve the right to "punt" and say "call us back once you've reproduced it on physical hardware".

Now, VMWare, to their credit, has a policy that they call "Total Ownership", where they will accept accountability for any Oracle-related issues. You can read their official policy at the link below.

http://www.vmware.com/support/policies/oracle-support.html

It is my understanding that, as part of the "Total Ownership" policy, VMware will reproduce the problem on physical hardware for the customer if Oracle decides that VMWare is the problem.

What about Licensing?

Part of the big problem I've always had with Oracle on VMWare is caused by Oracle's per-CPU licensing policy. My original understanding was that, if you have a total of 64 cores in your VMWare cluster, it didn't matter if you were only using 8 cores for Oracle. Oracle would tell you that you had to pay for 64 cores. The idea behind this is that you could, potentially, resize the virtual machine to suit certain needs. Maybe you need more horsepower during month end?

What I've since learned is that Oracle has a policy document (below) that talks about "soft" vs. "hard" partitioning.

http://www.oracle.com/us/corporate/pricing/partitioning-070609.pdf

What I've described above would fall under the concept of "soft partitioning". However, "hard partitioning" methodologies allow for a different approach. VMWare has (naturally) a nice document that explains their approach to implementing clusters that are in compliance with Oracle's licensing requirements.

http://www.vmware.com/files/pdf/techpaper/vmw-understanding-oracle-certification-supportlicensing-environments.pdf

From that document, pay particular attention to section 2.2. In that section (specifically Scenario B), they discuss DRS Host Affinity rules and VMWare CPU pinning. (emphasis mine)

2.2 Clusters: Fully Licensed Versus Partially Licensed Clusters

Scenario B: Partially Licensed Clusters

When a customer does not have enough Oracle application instances to justify creating a dedicated cluster for those applications, only a subset of the hosts in the cluster are licensed for the Oracle application. In this situation, the customer must be careful to restrict the movement of Oracle application instances and virtual machines to only those hosts that are licensed to run the product.

In this case, DRS Host Affinity rules can be used to appropriately restrict the movement of virtual machines within the cluster. DRS Host Affinity is a vSphere feature that enables you to ensure that your Oracle applications are restricted to move only between a subset of the hosts—that is, not all hardware in the cluster is “available” to the Oracle software. DRS Host Affinity is a clustering technology and is not a mechanism for soft or hard partitioning of the servers. As explained in section 2.1, using VMware CPU pinning to partially license a host is not currently recognized by Oracle as a “hard partitioning” mechanism that receives subsystem pricing. However, once you have fully licensed the host, you have the right to design your environment such that the Oracle workloads are free to run on the licensed hosts inside the cluster. At present, Oracle does not have any stated policy regarding clustering mechanisms or DRS Host Affinity. Customers can easily maiatain records for compliance purposes as explained in section 2.3.

The advantages of this approach are similar to the advantages achieved with a fully licensed cluster. Because customers are typically able to increase the utilization of licensed processors, they reduce license requirements. However, consolidation ratios tend to be lower, because advanced vSphere features can be employed only on a smaller subset of the hosts.

VMWare CPU pinning is a feature that (in my understanding) would allow you to say that a given VM would only use certain cores in a physical host. So, if you have a single host with 16 cores, you can "pin" a given VM to four of them. According to Oracle's partitioning document (and VMWare's document), you would still be required to pay for all 16 cores in the box. The basic logic here is that Oracle's licensing policy is based on the number of cores in a physical server. You can't license part of a box. Period. No exceptions.

On the other hand, DRS Host Affinity, is a way to pin a virtual machine to a given host (or collection of hosts) within a cluster. So, let's say that you have ten (10) 8-core physical hosts (total of 80 cores) in your VMWare cluster. Using DRS Host Affinity, youcould restrict your Oracle VMs to a subset of those physical hosts. For example, if you restricted your Oracle VMs to only five (5) of those physical hosts, VMWare's contention is that you would only have to license 40 cores.

I sould probably include the standard "IANAL" (I am not a lawyer) disclaimer. I'm also not a VMWare administrator. What I am is a DBA and an IT Geek. That's pretty much the limit of it.

Hopefully this provides some clarity on the issue.

For further reading on the subject, here are a couple of blog links that I used in my research:

http://blogs.vmware.com/apps/2012/11/update-on-virtualizing-oracle.html

http://longwhiteclouds.com/2012/07/21/fight-the-fud-oracle-licensing-and-support-on-vmware-vsphere/

– James

EBS R12.2 Online Patching Webcast

For those of you who are (like me) anxiously awaiting the release of R12.2, the Applications Technology Group (ATG) is hosting a live webcast to preview the Online Patching feature in R12.2.

Kevin Hudson, one of the lead architects of this particular feature, is the presenter for this webcast.  I had the pleasure of attending Kevin's session at Collaborate 2012 where he discussed this feature in-depth.  He is an excellent presenter and is very well-versed on this topic.  I'm sure this webcast will be worth your time.

This two hour webcast will take place on June 14 @ 8:00am (Pacific Standard Time).

Full details are on Steven Chan's blog at the link below.

ATG Live Webcast June 14: Technical Preview of EBS 12.2 Online Patching

-- James

Password-less Login Using SSH Pre-Shared Keys

Way back when I started working with Unix (otherwise known as "the olden days" or "days of yore"), one of the tricks we used was a concept known as "remote login" and the "Berkeley R commands". This was based on a number of things, most of them depending on either the /etc/hosts.equiv or the ${HOME}/.rhosts file to establish the trusting relationship. Configuring these would allow you the ability to do some really neat things. Among them, copying files from one host to another using a command like rcp /tmp/file user@remotehost:/tmp/file without being asked for a password. This made for some really neat scripting opportunities and made it much easier to manage multiple systems.

Unfortunately, the Berkeley "R" commands are notoriously insecure. The way that the trusting was done was based entirely on the username and hostname of the remote user on the remote host. Literally, you told the server to trust "jmorrow@remotehost.mydomain.com". The problem with this is that all that was required was knowledge of the trusting relationship. All you had to do was set up a machine named "remotehost.mydomain.com" and create a "jmorrow" user on it. Then you could go anywhere that that trusting relationship allowed.

Fortunately for us, the cool features that were introduced by the Berkeley "R" commands are implemented much more securely in the SSH protocol and toolset.

The SSH Protocol can use pre-shared keys to establish trusting relationships. In this case, each node has both a public and a private key. When the client talks to the server, the client offers a " key". The server, which maintains a list of trusted "public keys", then compares that key to it's database to determine if it actually trusts the client. If the client passes the test, then it is allowed in without any further challenge. This can be very useful for administrators, automated file transfer, also for scripting interactions between hosts. Note that this is not a "Machine A" trusts "Machine B" relationship. It is "user@machinea" trusts "user@machineb".

For the purposes of this article, the "server" is the node that you are logging into from the "client". So, the "server" is the one that is doing the trusting. The terms "server" and "client" refer only to the role being played by each component in the ssh communications session. I should also mention that Oracle Real Application Clusters (RAC) depends on this relationship as well.

Generate your public/private key pairs [Both Client and Server]

The server (user@host) needs to have one, and each client (user@host) that is being trusted needs to have one.

Execute these two commands (in a Unix/Linux environment) to create both your rsa and your dsa keys. You will be prompted for a location to store the files (typically under ${HOME}/.ssh), and for a passphrase. In all cases, it's ok to accept the defaults.

ssh-keygen -t rsa

ssh-keygen -t dsa

If you know you don't want to use a passphrase, you could generate the keys with these two commands:

ssh-keygen -t rsa -f ${HOME}/.ssh/id_rsa -N ""

ssh-keygen -t dsa -f ${HOME}/.ssh/id_dsa -N ""

Transfer the public key files from the client to the server

I prefer to make sure that I have a uniquely named copy of the public keys (makes it easier to transfer to another box when first establishing the relationship).

cd ${HOME}/.ssh

ls -1 id_[dr]sa.pub |while read LINE

do

cp ${LINE} ${LINE}.`whoami`@`hostname -s`

done

Now copy these files to the server:

scp ${LINE}.`whoami`@`hostname -s` trustinguser@trustingserver:.ssh/.

Copy the public keys you're trusting into the authorized_keys file

Here, we'll need to put those keys into the authorized_keys file. Do this for each of the files that you transferred in the previous step.

cd ${HOME}/.ssh

cat >> authorized_keys

Make sure permissions are correct

If the permissions on these files are too open, the trusting relationship will not work. Here are my recommendations:

chmod 600 ${HOME}/.ssh/auth*

chmod 700 ${HOME}/.ssh

chmod 644 ${HOME}/.ssh/id_[dr]sa.pub*

chmod 600 ${HOME}/.ssh/id_[dr]sa

Now, you should be able to ssh from the client to the server witout being prompted for a password.

– James

Adventures in SQL*Net encryption (E-Business Suite edition)

This is a strange problem that we encountered at one of my clients recently. Basically, they're turning on the SQL*Net encryption features of Oracle Advanced Networking Option in their R12.1.3/11.2.0.3 environments. To do this, we're following note 376700.1 "Enabling SSL in Oracle E-Business Suite Release 12". In particular, Section 10 of that document.

Everything was working well. E-Business Suite was able to connect just fine, even our developers were connecting through TOAD. The only significant problem we encountered was with the Windows-based Workflow Builder 2.6.3.5 client. Here, despite a number of attempts at playing with the sqlnet.ora settings, we were consistantly getting the "ORA-12599 TNS:cryptographic checksum mismatch" error.

The mandate from our internal security team was that all outside traffic had to be encrypted. Since we have an external "DMZ" appsTier node, that meant that we had to turn on this SQL*Net encryption. For ALL of our environments.

Naturally, we opened a SR. As time drug on, we started talking about other options. Do we create one non-production environment entirely behind the firewall for the developers to use that doesn't have the encryption turned on? What about making them ssh to the box directly and use WFLOAD?

Finally, after a couple of months of back-and-forth with the support analysts, plenty of "level 16" traces, and a couple of debates, support was able to supply a solution.

With ANO, you specify params like this in the sqlnet.ora file:

#

# Server Settings

#

SQLNET.ENCRYPTION_TYPES_SERVER=(AES256,AES192,3DES112,3DES168)

SQLNET.ENCRYPTION_SERVER=REQUIRED

SQLNET.CRYPTO_CHECKSUM_SERVER=REQUIRED

SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER=MD5

In the case of the "ENCRYPTION_TYPES_SERVER", this is a list of encryption algorithms that the server will accept. The server will negotiate the strongest encryption algorithm with the client.

On the client, you would have similar parameters in the sqlnet.ora file (well, you don't really need them because in many cases things will auto-negotiate based on defaults):

#

# Client Settings

#

SQLNET.ENCRYPTION_TYPES_CLIENT=(AES256,AES192,3DES112,3DES168)

SQLNET.ENCRYPTION_CLIENT=REQUIRED

SQLNET.CRYPTO_CHECKSUM_CLIENT=REQUIRED

SQLNET.CRYPTO_CHECKSUM_TYPES_CLIENT=MD5

Again, the ENCRYPTION_TYPES_CLIENT, this is a list of the algorithms that the client "can speak". Now, in the case of Workflow Builder 2.6.3.5 (which installs with a 10.1.0.2 client), it appears that the client is negotiating "in bad faith". Basically, it is allowing the negotiation to take place "normally". Based on that, the server will choose the strongest encryption from the ones common to both client and server. Unfortunately, the even though the client *says* it can handle AES256 (and others), it actually can't. So, for the Workflow Builder client, you are limited to the much older RC4_* encryption algorithms.

So, for Workflow Builder 2.6.3.5, here is what we have found to work for the client-side sqlnet.ora:

#

# Client Settings

#

SQLNET.ENCRYPTION_TYPES_CLIENT=rc4_256

SQLNET.ENCRYPTION_CLIENT=REQUIRED

SQLNET.CRYPTO_CHECKSUM_CLIENT=REQUIRED

SQLNET.CRYPTO_CHECKSUM_TYPES_CLIENT=MD5

You also need to make sure that the server-side sqlnet.ora contains the following line (or, at least contains the RC4_* encryption type you're intending to use):

SQLNET.ENCRYPTION_TYPES_SERVER=(AES256,AES192,3DES112,3DES168,RC4_256,RC4_128,RC4_56,RC4_40)

NOTE: In my testing, these algorithms work RC4_40, RC4_56, RC4_128, RC4_256 (we're using RC4_256 because it is the strongest of the RC4_* choices). The various RC_* encryption types are NOT mentioned in the sample sqlnet.ora provided by 376700.1 and, given their age, I would NOT recommend using them in a production environment.



– James