Friday, November 19, 2010

ORA-12641 "Authentication service failed to initialize" on multi homed machines

Update: the TMPDIR environment variable gets picked up; the file will be created in that directory. A technical note will be released, as this is not documented anywhere.
I have been struggeling to get Kerberos Authentication running on multi homes machines at my clients site. I just could not figure out why one package (HP-speak for virtual machines) works just fine, where the next errs off with a dreaded "ORA-12641: Authentication service failed to initialize".
Mind you - both using the same keytab and Kerberos configuration files, and similar sqlnet.ora files.

Go trace, young man, go trace!


Today I had the awakening moment to enable server tracing on the database side: added the following code to sqlnet.ora:
trace_level_server = 32
trace_file_server = svr_trc
trace_directory_client = $TNS_ADMIN/trace

Sure enough, comparing the package with errors against the one that validated me just fine using Kerberos, gave me some idea:

[19-NOV-2010 12:46:38:931] nauk5ra_rcinit: Validating existing replay cache.
[19-NOV-2010 12:46:38:931] snauk5g_open_file: entry
[19-NOV-2010 12:46:38:931] snauk5g_open_file: Setting write lock.
[19-NOV-2010 12:46:38:931] snauk5g_open_file: Opening /var/tmp/kerb5srv.RC.
[19-NOV-2010 12:46:38:931] snauk5k_lock_file: entry
[19-NOV-2010 12:46:38:931] snauk5k_lock_file: Setting exclusive lock.
[19-NOV-2010 12:46:38:931] snauk5k_lock_file: exit
[19-NOV-2010 12:46:38:931] snauk5g_open_file: exit
[19-NOV-2010 12:46:38:931] nauk5rz_validate: entry
[19-NOV-2010 12:46:38:931] nauk5rz_validate: exit
[19-NOV-2010 12:46:38:931] snauk5t_close_file: entry
[19-NOV-2010 12:46:38:931] snauk5k_lock_file: entry
[19-NOV-2010 12:46:38:931] snauk5k_lock_file: Resetting lock.
[19-NOV-2010 12:46:38:931] snauk5k_lock_file: exit
[19-NOV-2010 12:46:38:931] snauk5t_close_file: exit

Now, compare that to the faulty environment:

[19-NOV-2010 12:38:56:805] nauk5ra_rcinit: Validating existing replay cache.
[19-NOV-2010 12:38:56:805] snauk5g_open_file: entry
[19-NOV-2010 12:38:56:805] snauk5g_open_file: Setting write lock.
[19-NOV-2010 12:38:56:805] snauk5g_open_file: Opening /var/tmp/kerb5srv.RC.
[19-NOV-2010 12:38:56:805] snauk5g_open_file: open failed with 13: File I/O error
.
[19-NOV-2010 12:38:56:805] snauk5g_open_file: Returning 203: File not found
.
[19-NOV-2010 12:38:56:805] snauk5g_open_file: exit
[19-NOV-2010 12:38:56:805] nauk5ra_rcinit: Could not open cache.
[19-NOV-2010 12:38:56:805] nauk5ru_create: entry
[19-NOV-2010 12:38:56:805] nauk5ru_create: Creating replay cache.
[19-NOV-2010 12:38:56:806] snauk5g_open_file: entry

This will eventually end in

[19-NOV-2010 12:38:56:807] na_csrd: exit
[19-NOV-2010 12:38:56:807] nacomer: error 12641 received from authentication service

Workaround


Now, I would not write this, if I did not need to remind myself of the following. There is a workaround until this is permanently fixed by Oracle, but it is not without risk.
The problem is on file permissions. Once the file is created, it gets 600, or read-write for owner only permissions. Change that to 666 (read-write to everybody), and you're done.
In fact, the file does not need to have any contents:

rm /var/tmp/kerb5srv.RC
touch /var/tmp/kerb5srv.RC
chmod 666 /var/tmp/kerb5srv.RC

Asof now, all packages on this machine can use Kerberos authentication.
Please note: kerb5srv is the name of the Service Principal chosen by my client overhere - your implementation may use a different name. Use that (or, trace to see what file is used)

By the way, the entry on Privacy was updated, due to the signing of this.

The Catch


Apart from the fact you may have reservations to open up a file to the world like that, the file itself resides in a temporary directory, /var/tmp.
If any system administrator decides to clean that up, the dreaded ORA-12641 will be all over the place.
Just rerun code above, and you're done.

Permanent Solution


There seems no sqlnet variable to cure this; sqlnet.KERBEROS5_CC_NAME brings nothing. Strictly speaking, it is not the Credential Cache, but the service principal cache, so that figures.
What I would like to see is one of:
  • introduce an environment variable that allows to govern location of this file
  • use a suffix, just like in the credentials cache.
Update: the TMPDIR environment variable does point one...

Sunday, November 14, 2010

Congratulations, Sebastian

Sebastian Vettel, youngest world champion Formula 1 ever, congratulations!

And perhaps Ferrari should rethink contracting Mr. F. Alonso - the unsportsmanlike behavior towards Petrov says it all... If you cannot pass Petrov in like 20-odd rounds, you're clearly not World Champion material. There's only one to blame, Fernando - that is you. Not Petrov!

Thursday, November 04, 2010

Happy Days

Some post are still in backlog, I've been here...

Support pays off!

Update on a previous entry: we have a patch, and it actually works.
So when you ever get in the situation where fixed user database link do not work due to kerberos aare environments, ask for patch 8482256.
Available (and tested!) for HP-UX 11.23 and up; should become available for MS Windows (Server 2003, 32 bit and up), as well as VMS.

Monday, September 27, 2010

mount Oracle

Just curious:

mountd 20048/tcp NFS mount protocol
mountd 20048/udp NFS mount protocol
# Nicolas Williams <Nicolas.Williams_at_oracle.com> 09 August 2010

Sunday, September 26, 2010

ORA-12638: Status 82 "Not a bug"

One thing that completely spoils the previous entry on Kerberos authentication in the database, is the state of denial Oracle is in. It seems impossible to admit there's a bug in SQL*Net in the sense there's no fallback for username/password authentication when Kerberos fails.

ORA-12638 [Credential retrieval failed]

What's the case? Well, after you have set up Kerberos authentication, your database links seize working with ORA-12638: Credential retrieval failed, as soon as both databases are Kerberos aware.

Now, of course this is a situation you do not want. You want administrators and users to log in using their working environment credentials, but you rely on named accounts, and passwords, for database links.
Alas. Here comes Bug 6341668: CLIENT CAN NOT CONNECT TO DATABASE BY PASSWORD IF ANY KERBEROS PROBLEM OCCURS. Current status: 82 "Closed, not a bug". Al least someone else tried Kerberos authentication...
This is in contradiction with another Kerberos-related bug, # 2647883: Kerberos current user database links do not work (See Doc ID 2647883.8), where the actual workaround is ... use fixed user database links (!).
Catch-22, eh?

I hate it when I get confronted with this rubbish on Friday afternoons. It ruins my weekend!
Oh - might anyone wonder: this is cross platform, 10.2.0.4.

One other reference on Oracle Support: TNS-12638 when database authentication tried on Kerberos setup (Doc ID 972896.1)

Updates

ASO is standard

There's a note stating the Advanced Security Option is now standard with the Enterprise Database Install. Makes you wonder: is it included in the standard license fee, then?

Patch 10.2.0.5

Took the time to update the 10.2.0.4 environment I tested the scenario in to 10.2.0.5, which does have some Kerberos related bugs resolved. The result of all this is not only a ruined Sunday, it's also not resolving anything.
C:\>sqlplus /@db10
SQL*Plus: Release 10.2.0.5.0 - Production on Sun Sep 26 18:32:25 2010
Copyright (c) 1982, 2010, Oracle. All Rights Reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - Production
With the Partitioning, Data Mining and Real Application Testing options

SQL> select sysdate from dual@fvb;
select sysdate from dual@fvb
*
ERROR at line 1:
ORA-12638: Credential retrieval failed

Only after removing the services from sqlnet.ora on the server side, it works:

[ora10@db1 admin]$ cat sqlnet.ora
names.default_domain=home.local
# sqlnet.authentication_services=(beq, kerberos5)
sqlnet.kerberos5_conf=/oracle/db10/admin/krb5.conf
sqlnet.kerberos5_keytab=/oracle/db10/admin/db1.keytab
sqlnet.authentication_kerberos5_service=orcl
sqlnet.kerberos5_conf_mit=true

Which makes the same query on the client work... instantaneous!

SQL> select sysdate from dual@fvb;

SYSDATE
---------
26-SEP-10

It's getting weirder

I stumbled upon this, trying to find a workable solution on maintainting two TNS_ADMIN locations on the server. A similar problem has been logged as Bug 5054469: "DATABASE AUTHENTICATED USERS REQUIRE KERBEROS CREDENTIAL CACHE TO CONNECT", which has been closed as duplicate of (internal) bug 9056628.
Explain: how can a bug# 5 million-and-something be closed as a duplicate of bug# almost-twice-as-high? Makes me wonder.
And the best is this: Doc ID 972896.1 (TNS-12638 when database authentication tried on Kerberos setup) states "This has been an issue in the past that the authentication services are not skipped to next when first fails".
Yeah right: has been... in the past...

Wednesday, September 22, 2010

Synchronize OID 10.1.4.3 with Active Directory

There's quite a difference between 10.1.4.1 (and earlier versions) and 10.1.4.3 in the syncing business. You may now use multiple profiles quite easily and even have separate jobs for each profile. This would allow for a multi-AD-domain company to allow for different frequencies of synchronization: have one, well-know busy AD Domain synchronize very frequently, and other less frequent.
You may also enable or disable profiles by simply removing them from the running sync job(s); that process is called disassociation.

Receipe

So, what do I need?
1 Active Directory Installation
1 Oracle Identity Management Installation

I go both, one running VMWare (MS AD - I had it shipped as VMWare image), one Oracle, installed under VirtualBox.

Start it up

As the Oracle installation is as basic as it can be, it needs starting up:
frank@frank-cs03:~$ ssh oracle@192.168.1.222
oracle@192.168.1.222's password:
Last login: Fr Sep 10 16:43:24 2010
[oracle@oracleas ~]$ . oraenv
ORACLE_SID = [oracle] ? iasdb
[oracle@oracleas ~]$ lsnrctl start

LSNRCTL for Linux: Version 10.1.0.5.0 - Production on 29-DEC-2010 09:46:40

Copyright (c) 1991, 2004, Oracle. All rights reserved.

Starting /oracle/ias/10.1.4/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 10.1.0.5.0 - Production
System parameter file is /oracle/ias/10.1.4/network/admin/listener.ora
Log messages written to /oracle/ias/10.1.4/network/log/listener.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=oracleas)(PORT=1521)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracleas)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 10.1.0.5.0 - Production
Start Date 29-DEC-2010 09:46:41
Uptime 0 days 0 hr. 0 min. 0 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /oracle/ias/10.1.4/network/admin/listener.ora
Listener Log File /oracle/ias/10.1.4/network/log/listener.log
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=oracleas)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

I need a tcp based listener, because of the OID/ldap stack, that uses tcp. Otherwise, I would not have gone through all of the tcp stack, but use Direct Calls, or even better: bequeth, which does not need a listener at all. Anyway - open the database:

[oracle@oracleas ~]$ sqlplus / as sysdba

SQL*Plus: Release 10.1.0.5.0 - Production on Wed Dec 29 09:47:33 2010

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 281018368 bytes
Fixed Size 779000 bytes
Variable Size 229645576 bytes
Database Buffers 50331648 bytes
Redo Buffers 262144 bytes
Database mounted.
Database opened.
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.5.0 - Production
With the Partitioning, OLAP and Data Mining options

After all that, it is time to get ldap running. It needs to connect to the database, so this is the correct starting order:

[oracle@oracleas ~]$ $ORACLE_HOME/opmn/bin/opmnctl startall
opmnctl: starting opmn and all managed processes...

See if all is started:
[oracle@oracleas ~]$ $ORACLE_HOME/opmn/bin/opmnctl status

Processes in Instance: ias_1014.oracleas
-------------------+--------------------+---------+---------
ias-component | process-type | pid | status
-------------------+--------------------+---------+---------
DSA | DSA | N/A | Down
LogLoader | logloaderd | N/A | Down
dcm-daemon | dcm-daemon | N/A | Down
OC4J | OC4J_SECURITY | 2811 | Alive
HTTP_Server | HTTP_Server | 2726 | Alive
OID | OID | 2731 | Alive

If you would like to know which ports are used by the http server, use the "-l" option of the opmnctl command (-l stands for "long").
Now, with the OID, there's another check you ought to be aware of. Similar to the way the Oracle Process Manager (opmn) monitors, OID has it's own set of monitors. Activity of these can be checked by using the ldapcheck command:
[oracle@oracleas ~]$ $ORACLE_HOME/ldap/bin/ldapcheck

Checking Oracle Internet Directory Processes ...ALL

Process oidmon is Alive as PID 2731
Process oidldapd is Alive as PID 2749
Process oidldapd is Alive as PID 2783
Not Running ---- Process oidrepld
Process odisrv is Alive as PID 2770

oidmon (PID 2731) is the OID Monitoring process itself; it is the process, that is actually monitored by opmn. The two oidldapd processes are two LDAP Daemons. Other than that, there's an inactive Replication process, oidrepld, that is only used in Enterprise-class setups, as I described here.
The final process is the one, I'm interested in: it's the Oracle Directory Integration service, odisrv for short.

What if I have no ldapcheck?

Well, I did not, either. You can download diptester from here, unzip it, and rename "ldapcheck_for_HPUX" to ldapcheck, do a chmod 740, and you're in business.

What id ldapcheck returns
ps: illegal option -- o

Apply same medicin as "What if I do not have ldapcheck"

Synchronize!

Now, in order to get OID to synchronize from AD, I need a few things:
  • a highly privileged account on Active Directory
  • Names, or IP-addresses, of servers involved
  • an understanding of mapping, with base of OID as well as AD

As for the account, I'll use my own account, which is member of the administrators group - do not use this in production! But you do need an account, that is allowed to query the "Deleted users" section - and that requires quite some privileges. If you cannot see "Deleted Users", users can be removed from AD, but will always remain in OID. And that is a security issue...

IP addresses: The oracle iAS runs is called oracleas.home.local at 192.168.1.222, the PDC is located at 192.168.1.233.

Mapping: well, whoever invented that should be prosecuted. As long as you keep it basic, and simple, there are few problems. As soon as you company is operating multiple domains, it is a RPITA to get things running smoothly.

Profiles

Mapping is based on profiles. The fastest way to get things running is to take a copy of $ORACLE_HOME/ldap/odi/conf/activechg.map.master and use that as basis.
The base in OID is dc=bortel,dc=home, as you can see from this picture of the OID Aministration tool, oidadmin:

This is the correct entry, as this entry holds the users. Same is true for Active Directory, although I know no other way than to query the darn thing:

C:\Documents and Settings\frank>ldapsearch -Z -h w2k-pdc -p 3268 cn=frank cn
CN=frank,CN=Users,DC=home,DC=local
cn=frank
What I am doing here, is query my own credentials, using the built-in security (-Z), the localhost as ldap host (-h w2k-pdc), and query the Global Catalog port (-p 3268).
Anyway - my AD mapping is dc=home, dc=local.

Mapping rules

In the mapping file, this results in the first line being:

cn=users,dc=home,dc=local:cn=users,dc=bortel,dc=home

In words: maps everything, found under cn=users,dc=home,dc=local (in Active Directory, as it is left of the colon) to cn=users,dc=bortel,dc=home (in Oracle Internet Directory, as it is right of the colon).

These are the domain rules.

If you have more domains, just add mapping rules. If you have organized your domain in Units (Organizational Units, or "ou" in LDAP-speak), start mapping rules at that level, and add, or omit to your needs.

dipadmin

Now, a new tool needs to be used, the dispassistant. User is dipadmin - password of dipadmin is identical to the password of orcladmin:


New in this version is the use of Connector Profiles, and profile Sets.
Due to badly documented features, and some documentation bugs it is best to create a new Connector Group.
First of all, rename the Connector Groups "configset1" to "standardgroup". The original name will just create misunderstanding, as the are configsets in use with OID.


Then, create the group you will be using for your actions - the "standardgroup" will remain as the group holding all defaults, but will not actually be used. My new group is called "demo":


Next, go to the standardgroup, and disassociate ActiveChgImp:


Then, select the newly created group "demo", and associate the profile:


Note, the profile is still Disabled. Edit the profile, and fill in the connection defaults on the first tab:


On the third tab, make sure the filter properties do not have quotes around them - it's a bug:


On the last tab, I always change "Continue on Error" to Yes; I have no tools to monitor whether every change gets into OID correctly. Most of the errors I see, are on duplicated entries anyway.
You may also want to change the frequency of synchronization: every 60 seconds seems on the high side, escpecially if you have 7 profiles, like my customer, one of which connects to a Active Directory with 13,000 users.


Modify the profile mapping file to your liking, and load it into the profile:

[oracle@oracleas ~]$ dipassistant mp -D cn=orcladmin -profile=activechgimp odip.profile.mapfile=demo_user.map
Password:
Profile successfully modified.
[oracle@oracleas ~]$

Bootstrap


The first, initial (bulk) load of the accounts is call bootstrapping:

[oracle@oracleas ~]$ dipassistant mp -D cn=orcladmin -profile=activechgimp odip.profile.mapfile=demo_user.map
Password:
Profile successfully modified.
[oracle@oracleas ~]$

Results of this process may be found in $ORACLE_HOME/ldap/odi/log/bootstrap.log and ~.trc

You can also inspect the OID with the oidadmin tool:


(and see there's a problem with krbPrincipalName...)

Activate

Now, all that's left is to start the process of syncs every x seconds. In order to do that, I'll need to start the connectorgroup:

[oracle@oracleas ~]$ $ORACLE_HOME/ldap/bin/ldapcheck

Checking Oracle Internet Directory Processes ...ALL

Process oidmon is Alive as PID 2513
Process oidldapd is Alive as PID 2519
Process oidldapd is Alive as PID 2526
Not Running ---- Process oidrepld
Process odisrv is Alive as PID 2521
[oracle@oracleas ~]$ oidctl connect=iasdb server=odisrv instance=2 configset=1 flags="grpid=demo" start
NLS_LANG not set in environment
Setting NLS_LANG to AMERICAN_AMERICA.AL32UTF8
oidctl:Waiting for oidmon to start ODISRV (instance=2)
oidctl:Waiting for oidmon to start ODISRV (instance=2)
oidctl:Started ODISRV (instance=2) with PID : 21496 successfully

[oracle@oracleas ~]$ $ORACLE_HOME/ldap/bin/ldapcheck

Checking Oracle Internet Directory Processes ...ALL

Process oidmon is Alive as PID 2513
Process oidldapd is Alive as PID 2519
Process oidldapd is Alive as PID 2526
Not Running ---- Process oidrepld
Process odisrv is Alive as PID 2521
Process odisrv is Alive as PID 21496
[oracle@oracleas ~]$

There are two odi processes now. The second one is the AD-OID synchronisation.
Logging of this process is in the same directory as the bootstrap: $ORACLE_HOME/ldap/odi/log.
You will see the timestamp changing (and the file will grow.

Now, after adding a new user in AD, I can quey these:

[oracle@oracleas ~]$ ldapsearch cn=frank* uid
cn=frank,cn=users,dc=bortel,dc=home

cn=frank w.j.. van bortel,cn=users,dc=bortel,dc=home
uid=frank2@home.local


And that proves, the synchronisation is working!

Groups, and all that


This is to be added: how to sync groups, and split these. I will also mention some considerations about syncing, especially on multiple domains.