ORA-12641 "Authentication service failed to initialize" on multi homed machines

Update: the TMPDIR environment variable gets picked up; the file will be created in that directory. A technical note will be released, as this is not documented anywhere.
I have been struggeling to get Kerberos Authentication running on multi homes machines at my clients site. I just could not figure out why one package (HP-speak for virtual machines) works just fine, where the next errs off with a dreaded "ORA-12641: Authentication service failed to initialize".
Mind you - both using the same keytab and Kerberos configuration files, and similar sqlnet.ora files.

Go trace, young man, go trace!

Today I had the awakening moment to enable server tracing on the database side: added the following code to sqlnet.ora:

trace_level_server = 32
trace_file_server = svr_trc
trace_directory_client = $TNS_ADMIN/trace

Sure enough, comparing the package with errors against the one that validated me just fine using Kerberos, gave me some idea:

[19-NOV-2010 12:46:38:931] nauk5ra_rcinit: Validating existing replay cache.
[19-NOV-2010 12:46:38:931] snauk5g_open_file: entry
[19-NOV-2010 12:46:38:931] snauk5g_open_file: Setting write lock.
[19-NOV-2010 12:46:38:931] snauk5g_open_file: Opening /var/tmp/kerb5srv.RC.
[19-NOV-2010 12:46:38:931] snauk5k_lock_file: entry
[19-NOV-2010 12:46:38:931] snauk5k_lock_file: Setting exclusive lock.
[19-NOV-2010 12:46:38:931] snauk5k_lock_file: exit
[19-NOV-2010 12:46:38:931] snauk5g_open_file: exit
[19-NOV-2010 12:46:38:931] nauk5rz_validate: entry
[19-NOV-2010 12:46:38:931] nauk5rz_validate: exit
[19-NOV-2010 12:46:38:931] snauk5t_close_file: entry
[19-NOV-2010 12:46:38:931] snauk5k_lock_file: entry
[19-NOV-2010 12:46:38:931] snauk5k_lock_file: Resetting lock.
[19-NOV-2010 12:46:38:931] snauk5k_lock_file: exit
[19-NOV-2010 12:46:38:931] snauk5t_close_file: exit

Now, compare that to the faulty environment:

[19-NOV-2010 12:38:56:805] nauk5ra_rcinit: Validating existing replay cache.
[19-NOV-2010 12:38:56:805] snauk5g_open_file: entry
[19-NOV-2010 12:38:56:805] snauk5g_open_file: Setting write lock.
[19-NOV-2010 12:38:56:805] snauk5g_open_file: Opening /var/tmp/kerb5srv.RC.
[19-NOV-2010 12:38:56:805] snauk5g_open_file: open failed with 13: File I/O error
.
[19-NOV-2010 12:38:56:805] snauk5g_open_file: Returning 203: File not found
.
[19-NOV-2010 12:38:56:805] snauk5g_open_file: exit
[19-NOV-2010 12:38:56:805] nauk5ra_rcinit: Could not open cache.
[19-NOV-2010 12:38:56:805] nauk5ru_create: entry
[19-NOV-2010 12:38:56:805] nauk5ru_create: Creating replay cache.
[19-NOV-2010 12:38:56:806] snauk5g_open_file: entry

This will eventually end in

[19-NOV-2010 12:38:56:807] na_csrd: exit
[19-NOV-2010 12:38:56:807] nacomer: error 12641 received from authentication service

Workaround

Now, I would not write this, if I did not need to remind myself of the following. There is a workaround until this is permanently fixed by Oracle, but it is not without risk.
The problem is on file permissions. Once the file is created, it gets 600, or read-write for owner only permissions. Change that to 666 (read-write to everybody), and you're done.
In fact, the file does not need to have any contents:

rm /var/tmp/kerb5srv.RC
touch /var/tmp/kerb5srv.RC
chmod 666 /var/tmp/kerb5srv.RC

Asof now, all packages on this machine can use Kerberos authentication.
Please note: kerb5srv is the name of the Service Principal chosen by my client overhere - your implementation may use a different name. Use that (or, trace to see what file is used)

By the way, the entry on Privacy was updated, due to the signing of this.

The Catch

Apart from the fact you may have reservations to open up a file to the world like that, the file itself resides in a temporary directory, /var/tmp.
If any system administrator decides to clean that up, the dreaded ORA-12641 will be all over the place.
Just rerun code above, and you're done.

Permanent Solution

There seems no sqlnet variable to cure this; sqlnet.KERBEROS5_CC_NAME brings nothing. Strictly speaking, it is not the Credential Cache, but the service principal cache, so that figures.
What I would like to see is one of:

• introduce an environment variable that allows to govern location of this file
• use a suffix, just like in the credentials cache.

Update: the TMPDIR environment variable does point one...

Kerberos errors

As extension of the previous blog on Windows Native Authentication with Oracle, this little piece of info:

Kerberos Error 68.
Kerberos testing (kinit -k -t command) responded with

kinit: KRB5 error code 68 while getting initial credentials

Searches revealed:

KDC_ERR_WRONG_REALM 68 Reserved for future use
is being returned by Active Directory because your users are attempting to obtain a Kerberos TGT for a realm that is not hosted on the server to which they are authenticating.
The existing MIT Kerberos distribution that you are using does not know how to respond to this error. Windows machines can attempt to search the Active Directory Global Catalog in order to determine the actual principal name to use for authentication.

The krb5.conf file had port 88 specified on (one of the member) Active Directory server. Changing that to port 3268 (which is the Global Catalog port), changes the error into this:

kinit: Cannot contact any KDC for requested realm while getting initial credentials

I think this means the realm (domain in AD speak) is not serviced by this server. Problem is: where is it serviced.
Addition.
OK - got that solved; you can specify many Kerberos servers in the [realms] section of the krb5.conf file. Doing so resolved the issue of error 68.

Kerberos Encryption
Now, the next problem arises:

kinit: Bad encryption type while getting initial credentials

klist
There is a handy utility, klist, that can help out here. Klist can read the keytab file, and display all kinds of details, one of which is the encryption type used. Previous keytab files revealed RSA-MD5 was used, the latest one revealed CRC32:

klist -k -e -K -t FILE:/home/bortel/second.keytab

Keytab name: FILE:/home/bortel/second.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
1 01/01/70 01:00:00 HTTP/[nondisclosed] (DES cbc mode with CRC-32) (0x3e4986bc07972cda)


Keytab name: FILE:/home/bortel/first.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
4 01/01/70 01:00:00 HTTP/[nondisclosed] (DES cbc mode with RSA-MD5) (0x855d98e6793186e9)

With the first keytab file (listed as second entry), WNA works without any changes from the Oracle examples. The second keytab file (listed on top) has a different encription type, compared to the first. This might explain the encryption error...
Sure enough; altering the krb5.conf file, adding enctypes, so that the file reads the following resolved that issue:

[libdefaults]
default_realm = HOME.LOCAL
default_tkt_enctypes = des-cbc-crc
default_tgs_enctypes = des-cbc-crc
clockskew = 300

[realms]

Another enctype would be des-cbc-md5. This looks like the default one, as I did not specify enctypes in an earlier krb5.conf file.

Windows 2000 versus Windows 2003?
Now for the underlying reason, I can only guess. Is this a MS Windows issue? Did MS change from des-cbc-crc to des-cbc-md5 between Windows 2000 Server and Windows Server 2003? Seems unlikely, unless MS Windows always tries CRC32 as well as MD5.

Anyway, the problems I was facing were resolved, as this shows:

kinit -k -t /home/bortel/second.keytab HTTP/[nondisclosed]
klist
Ticket cache: /tmp/krb5cc_879
Default principal: HTTP/[nondisclosed]@HOME.LOCAL

Valid starting Expires Service principal
01/30/08 09:39:37 01/30/08 19:39:37 krbtgt/HOME.LOCAL@HOME.LOCAL

klist also allows to show the encryption type used:

klist -e
Ticket cache: /tmp/krb5cc_879
Default principal: HTTP/[nondisclosed]@HOME.LOCAL

Valid starting Expires Service principal
01/30/08 09:39:37 01/30/08 19:39:37 krbtgt/HOME.LOCAL@HOME.LOCAL
Etype (skey, tkt): DES cbc mode with CRC-32, DES cbc mode with CRC-32