One thing that completely spoils the previous entry on Kerberos authentication in the database, is the state of denial Oracle is in. It seems impossible to admit there's a bug in SQL*Net in the sense there's no fallback for username/password authentication when Kerberos fails.
ORA-12638 [Credential retrieval failed]
What's the case? Well, after you have set up Kerberos authentication, your database links seize working with ORA-12638: Credential retrieval failed, as soon as both databases are Kerberos aware.
Now, of course this is a situation you do not want. You want administrators and users to log in using their working environment credentials, but you rely on named accounts, and passwords, for database links. Alas. Here comes Bug 6341668: CLIENT CAN NOT CONNECT TO DATABASE BY PASSWORD IF ANY KERBEROS PROBLEM OCCURS. Current status: 82 "Closed, not a bug". Al least someone else tried Kerberos authentication... This is in contradiction with another Kerberos-related bug, # 2647883: Kerberos current user database links do not work (See Doc ID 2647883.8), where the actual workaround is ... use fixed user database links (!). Catch-22, eh?
I hate it when I get confronted with this rubbish on Friday afternoons. It ruins my weekend! Oh - might anyone wonder: this is cross platform, 10.2.0.4.
One other reference on Oracle Support: TNS-12638 when database authentication tried on Kerberos setup (Doc ID 972896.1)
ASO is standard
There's a note stating the Advanced Security Option is now standard with the Enterprise Database Install. Makes you wonder: is it included in the standard license fee, then?
Took the time to update the 10.2.0.4 environment I tested the scenario in to 10.2.0.5, which does have some Kerberos related bugs resolved. The result of all this is not only a ruined Sunday, it's also not resolving anything.
C:\>sqlplus /@db10 SQL*Plus: Release 10.2.0.5.0 - Production on Sun Sep 26 18:32:25 2010 Copyright (c) 1982, 2010, Oracle. All Rights Reserved.
Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - Production With the Partitioning, Data Mining and Real Application Testing options
SQL> select sysdate from dual@fvb; select sysdate from dual@fvb * ERROR at line 1: ORA-12638: Credential retrieval failed
Only after removing the services from sqlnet.ora on the server side, it works:
Which makes the same query on the client work... instantaneous!
SQL> select sysdate from dual@fvb;
SYSDATE --------- 26-SEP-10
It's getting weirder
I stumbled upon this, trying to find a workable solution on maintainting two TNS_ADMIN locations on the server. A similar problem has been logged as Bug 5054469: "DATABASE AUTHENTICATED USERS REQUIRE KERBEROS CREDENTIAL CACHE TO CONNECT", which has been closed as duplicate of (internal) bug 9056628. Explain: how can a bug# 5 million-and-something be closed as a duplicate of bug# almost-twice-as-high? Makes me wonder. And the best is this: Doc ID 972896.1 (TNS-12638 when database authentication tried on Kerberos setup) states "This has been an issue in the past that the authentication services are not skipped to next when first fails". Yeah right: has been... in the past...
There's quite a difference between 10.1.4.1 (and earlier versions) and 10.1.4.3 in the syncing business. You may now use multiple profiles quite easily and even have separate jobs for each profile. This would allow for a multi-AD-domain company to allow for different frequencies of synchronization: have one, well-know busy AD Domain synchronize very frequently, and other less frequent. You may also enable or disable profiles by simply removing them from the running sync job(s); that process is called disassociation.
So, what do I need? 1 Active Directory Installation 1 Oracle Identity Management Installation
I go both, one running VMWare (MS AD - I had it shipped as VMWare image), one Oracle, installed under VirtualBox.
Start it up
As the Oracle installation is as basic as it can be, it needs starting up:
TNSLSNR for Linux: Version 10.1.0.5.0 - Production System parameter file is /oracle/ias/10.1.4/network/admin/listener.ora Log messages written to /oracle/ias/10.1.4/network/log/listener.log Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=oracleas)(PORT=1521))) Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracleas)(PORT=1521))) STATUS of the LISTENER ------------------------ Alias LISTENER Version TNSLSNR for Linux: Version 10.1.0.5.0 - Production Start Date 29-DEC-2010 09:46:41 Uptime 0 days 0 hr. 0 min. 0 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /oracle/ias/10.1.4/network/admin/listener.ora Listener Log File /oracle/ias/10.1.4/network/log/listener.log Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=oracleas)(PORT=1521))) (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521))) Services Summary... Service "PLSExtProc" has 1 instance(s). Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service... The command completed successfully
I need a tcp based listener, because of the OID/ldap stack, that uses tcp. Otherwise, I would not have gone through all of the tcp stack, but use Direct Calls, or even better: bequeth, which does not need a listener at all. Anyway - open the database:
[oracle@oracleas ~]$ sqlplus / as sysdba
SQL*Plus: Release 10.1.0.5.0 - Production on Wed Dec 29 09:47:33 2010
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup ORACLE instance started.
Total System Global Area 281018368 bytes Fixed Size 779000 bytes Variable Size 229645576 bytes Database Buffers 50331648 bytes Redo Buffers 262144 bytes Database mounted. Database opened. SQL> exit Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.5.0 - Production With the Partitioning, OLAP and Data Mining options
After all that, it is time to get ldap running. It needs to connect to the database, so this is the correct starting order:
[oracle@oracleas ~]$ $ORACLE_HOME/opmn/bin/opmnctl startall opmnctl: starting opmn and all managed processes...
See if all is started:
[oracle@oracleas ~]$ $ORACLE_HOME/opmn/bin/opmnctl status
Processes in Instance: ias_1014.oracleas -------------------+--------------------+---------+--------- ias-component | process-type | pid | status -------------------+--------------------+---------+--------- DSA | DSA | N/A | Down LogLoader | logloaderd | N/A | Down dcm-daemon | dcm-daemon | N/A | Down OC4J | OC4J_SECURITY | 2811 | Alive HTTP_Server | HTTP_Server | 2726 | Alive OID | OID | 2731 | Alive
If you would like to know which ports are used by the http server, use the "-l" option of the opmnctl command (-l stands for "long"). Now, with the OID, there's another check you ought to be aware of. Similar to the way the Oracle Process Manager (opmn) monitors, OID has it's own set of monitors. Activity of these can be checked by using the ldapcheck command:
Checking Oracle Internet Directory Processes ...ALL
Process oidmon is Alive as PID 2731 Process oidldapd is Alive as PID 2749 Process oidldapd is Alive as PID 2783 Not Running ---- Process oidrepld Process odisrv is Alive as PID 2770
oidmon (PID 2731) is the OID Monitoring process itself; it is the process, that is actually monitored by opmn. The two oidldapd processes are two LDAP Daemons. Other than that, there's an inactive Replication process, oidrepld, that is only used in Enterprise-class setups, as I described here. The final process is the one, I'm interested in: it's the Oracle Directory Integration service, odisrv for short.
What if I have no ldapcheck?
Well, I did not, either. You can download diptester from here, unzip it, and rename "ldapcheck_for_HPUX" to ldapcheck, do a chmod 740, and you're in business.
What id ldapcheck returns
ps: illegal option -- o
Apply same medicin as "What if I do not have ldapcheck"
Now, in order to get OID to synchronize from AD, I need a few things:
a highly privileged account on Active Directory
Names, or IP-addresses, of servers involved
an understanding of mapping, with base of OID as well as AD
As for the account, I'll use my own account, which is member of the administrators group - do not use this in production! But you do need an account, that is allowed to query the "Deleted users" section - and that requires quite some privileges. If you cannot see "Deleted Users", users can be removed from AD, but will always remain in OID. And that is a security issue...
IP addresses: The oracle iAS runs is called oracleas.home.local at 192.168.1.222, the PDC is located at 192.168.1.233.
Mapping: well, whoever invented that should be prosecuted. As long as you keep it basic, and simple, there are few problems. As soon as you company is operating multiple domains, it is a RPITA to get things running smoothly.
Mapping is based on profiles. The fastest way to get things running is to take a copy of $ORACLE_HOME/ldap/odi/conf/activechg.map.master and use that as basis. The base in OID is dc=bortel,dc=home, as you can see from this picture of the OID Aministration tool, oidadmin: This is the correct entry, as this entry holds the users. Same is true for Active Directory, although I know no other way than to query the darn thing:
What I am doing here, is query my own credentials, using the built-in security (-Z), the localhost as ldap host (-h w2k-pdc), and query the Global Catalog port (-p 3268). Anyway - my AD mapping is dc=home, dc=local.
In the mapping file, this results in the first line being:
In words: maps everything, found under cn=users,dc=home,dc=local (in Active Directory, as it is left of the colon) to cn=users,dc=bortel,dc=home (in Oracle Internet Directory, as it is right of the colon).
These are the domain rules.
If you have more domains, just add mapping rules. If you have organized your domain in Units (Organizational Units, or "ou" in LDAP-speak), start mapping rules at that level, and add, or omit to your needs.
Now, a new tool needs to be used, the dispassistant. User is dipadmin - password of dipadmin is identical to the password of orcladmin:
New in this version is the use of Connector Profiles, and profile Sets. Due to badly documented features, and some documentation bugs it is best to create a new Connector Group. First of all, rename the Connector Groups "configset1" to "standardgroup". The original name will just create misunderstanding, as the are configsets in use with OID.
Then, create the group you will be using for your actions - the "standardgroup" will remain as the group holding all defaults, but will not actually be used. My new group is called "demo":
Next, go to the standardgroup, and disassociate ActiveChgImp:
Then, select the newly created group "demo", and associate the profile:
Note, the profile is still Disabled. Edit the profile, and fill in the connection defaults on the first tab:
On the third tab, make sure the filter properties do not have quotes around them - it's a bug:
On the last tab, I always change "Continue on Error" to Yes; I have no tools to monitor whether every change gets into OID correctly. Most of the errors I see, are on duplicated entries anyway. You may also want to change the frequency of synchronization: every 60 seconds seems on the high side, escpecially if you have 7 profiles, like my customer, one of which connects to a Active Directory with 13,000 users.
Modify the profile mapping file to your liking, and load it into the profile:
Checking Oracle Internet Directory Processes ...ALL
Process oidmon is Alive as PID 2513 Process oidldapd is Alive as PID 2519 Process oidldapd is Alive as PID 2526 Not Running ---- Process oidrepld Process odisrv is Alive as PID 2521 [oracle@oracleas ~]$ oidctl connect=iasdb server=odisrv instance=2 configset=1 flags="grpid=demo" start NLS_LANG not set in environment Setting NLS_LANG to AMERICAN_AMERICA.AL32UTF8 oidctl:Waiting for oidmon to start ODISRV (instance=2) oidctl:Waiting for oidmon to start ODISRV (instance=2) oidctl:Started ODISRV (instance=2) with PID : 21496 successfully
Checking Oracle Internet Directory Processes ...ALL
Process oidmon is Alive as PID 2513 Process oidldapd is Alive as PID 2519 Process oidldapd is Alive as PID 2526 Not Running ---- Process oidrepld Process odisrv is Alive as PID 2521 Process odisrv is Alive as PID 21496 [oracle@oracleas ~]$
There are two odi processes now. The second one is the AD-OID synchronisation. Logging of this process is in the same directory as the bootstrap: $ORACLE_HOME/ldap/odi/log. You will see the timestamp changing (and the file will grow.
Now, after adding a new user in AD, I can quey these:
After introducing Kerberos, and getting SSO to the database working, it's time for some excitement: combine Single Sign On (SSO) with Enterprise User Security (EUS). Sure, I blogged about Enterprise Security a long time ago (See the "Enterprise Security" entries of 2005), but there I used certificates; in fact another identity. In this entry, some of the 2005 stuff returns, but parts will differ.
The latest and the greatest.
For those of you how wonder why I bother to blog about this "ancient" version, read the statement of direction on SSO server (note 979951.1 on MOS). Basically, when you use Portal, Forms or Discoverer, you need OSSO 10GR3 (even Forms 11G). It's not that ancient when it will be around for some time...
What I want, is the same a the previous entry on SSO, but I do not wish to create users: I want to use EUS instead.
In addition to what's already in place, I need a complete SSO infrastructure. The brave may want to do this Enterprise-sized setup, for now I'll just go with the "all-in-one" Identity Management install. Even patched it to 10.1.4.3.
In this entry, I'll demonstrate how easy it is to do Kerberos authentication to the database, or -rephrasing that- how to do WNA in the database. The goal of this exercise will be to allow users to log on to a remote database using the following syntax:
I like to think major sites have serious machines and ditto operating systems. Workstations are uncannily often based on MS Windows, so I'll use MS Windows for a client, and Linux for database server.
Both client and server need to have the Advanced Security Option installed, for which Oracle wants to get paid. Just read the license conditions: if you want to do Kerberos, you need to pay.
Now, how do you check whether you have that installed? On *ixes, you can use the command adapters:
According to the documentation, that is proof. I have found that it is not always the case, and you may find yourself without the Oracle Kerberos utilities. Better check if these are available, they sould be:
[ora10@db1 ~]$ ls $ORACLE_HOME/bin/ok* /oracle/db10/10GRel2/bin/okdstry /oracle/db10/10GRel2/bin/okinit /oracle/db10/10GRel2/bin/oklist /oracle/db10/10GRel2/bin/okdstry0 /oracle/db10/10GRel2/bin/okinit0 /oracle/db10/10GRel2/bin/oklist0
That concludes checks on the software stack on the server. If you miss anything: install the Advanced Security Option!
For MS Windows clients, there is unfortunately not much else to do than to fire up OUI, and list the installed options. Fore mentioned Kerberos utilities should also be available on the client, so you may want to check that out as well.
First thing to do, is to allow the database server to get Kerberos tickets. In order to do that, I need a Kerberos service principal. In order to do that, I need to create a domain user on my MS Windows 2000 Domain Controller: I have chosen the name of the instance, but that does not really matter. You may want to use a more generic user name if you have an environment with many instances per machine. Make sure you set "Use DES encryption" and "Do not require preauthentication", as shown here: You may have to patch your MicroSoft Active Directory to support this. Check out Knowledge Base article 833708 on how to do this in the registry, or apply SP 1. I think everybody has SP1 (or better) - I did not have to change anything on my PDC image, but this may cause problems.
Now that I created a user, I must transfer the credentials to the database server. That involves exporting keys to a file, and it is done using this statement on the PDC (or member domain controller):
The beauty is, that you may forget the password: you will never need it again. In fact, you need not know the password, unless things don't work, and you want to troubleshoot. You should remember the service name: you need to specify that in configuration files. I use "orcl" in this example - you may want to use a more generic name, like krb5srv. Oracle MOS Document states you can only use -crypto des-cbc-crc for encrytion; this is incomplete; you can also use -crypto des-cbc-md5, as I did. (you need md5 for HTTP principals, but that's an other story).
Binary transfer the keytab file to the server. These files are called keytab files, as they contain a table of keys - the keytab for short. Logon to the database server an cd to your TNS_ADMIN directory. Which happens to be $ORACLE_HOME/network/admin, if you do not use $TNS_ADMIN. Alter the sqlnet.ora file to look like:
Note the last line: it is MS specific, and allows the use of the MS Windows internal Kerberos cache, instead of a file based one (as specified the line above). Note however, using the internal cache will disallow the use of Oracle Kerberos utilities; these will err off with "OSD error"
The Kerberos configuration file is similar to the one on the server, and connects realms to domains:
You may now test the Kerberos setup. You should be able to get a ticket from the Kerberos Ticket Granting Server (which is MS Active Directory, in this case). Try to get a ticket for a known MS domain account, I can log on to the domain "home.local" as frank, that would make my account "frank@HOME.LOCAL" - and be careful, Kerberos is strangely case sensitive! The oklist utility reveals:
As much as I like privacy (google changed it's privacy statement just now, btw), I hate any form of censorship on the internet. RIM seems to give up on privacy, by allowing governments to decrypt Blackberry traffic. Google is not just content with our search and surf behaviour on the internet, it now wants to know what we're talking about and introduces free phone services. And Apple is censoring ping.
Sigh. What a world. Next entry will be on Oracle, and SSO to the database. Update: Craigslist does censor, too.