Sunday, June 10, 2007

High performance, High availablity in Oracle Application Server

Last update: Aug, 10, 10:53 (11g Download!)Ambitious?
Here is what I want to do. I have done setups according to the Enterprise Deployment Guide, ending up with a configuration similar to what you can see here, and I've also witnessed stuff, described in numerous metalink articles (so it must be hard :) ) and in the High Availability Guide, that lead to this.
Now, for some reason, my current assignment does not allow RAC setups. Reasoning fails, but I have given up after a year-and-a-half: "This is the first release of RAC - we don't do first releases". Which is crap, of course , since the predecessors have been around some 8 (yes - eight) releases: 7.1, 7.2, 7,3, 8.0, 8.1, 9.0, 9.1, 10.1 and now 10.2. Thanks to Oracle Marketing...
Anyway, that leads to a point where the database has become the single point of failure, when using the Enterprise setup. This setup uses the same (Clustered) database as OID storage, as well as the Application Server Repository.
Using the High Availability Guide, you will not have High Performance: as one link in the chain breaks, the whole chain is unavailable - you will still have the parallel chain, so availability does not suffer, just performance. This is due to the fact the databases do act as backups for OID, but not for the Application Server Metadata!
So, where the first setup is clustered on Application Server level, the second is not. Where the second setup allows one chain to become completely, or partially unavailable, the first approach will fail in the database department (which is not, and cannot be RAC!). What I want is best of both! I want Application Server Clustering, and Load Balancing, and Replication and Fail-Over! So, there you have it.
Ambitious? Sure!
Can it be done? Well, I actually don't know. You are here to find out.
When do you know? Well, at the end of the story, and this may become a lengthy one. I do not want to split is, as I did with the Enterprise Security entries, so I will update this article as I go.

Preparations
I have to my disposal three machines, all equipped with two harddisks. The latest replacement is equipped with an Intel E6600 processor, 4GB ram and two 320GiB SATA disks. It replaced the AMD 2100+ with 1.5GB memory and two 80GB ATA100 disks in a stripe set.
For test purposes, I already had a "server", which has been used before.
Both machines have been rebuilt, using the previous post.
All three machines are interconnected via a gigabit switch, using proper, short, 1GB certified network cables. The gigabit ethernet interfaces are the onboard ones, and price of the switch is not an issue anymore.
So much for the hardware; as for the software, better start early, as there is some 6,832MB (well over 6GB!) to be downloaded! Of course, the 400MB from CentOS is already completed. That leaves:
  • Downloads from oracle:
  1. 10G release2 Database (640MB for the Windows version)
  2. 10G Release2 Companion (another 640MB for the Windows version)
  3. Patch 10.2.0.3 (almost 900MB for Windows)
  4. CPU Apr2007 for the database (another 140MB)
  5. iAS 10G release 2 (2 GB in 4 cpio files)
  6. iAS 10G release 2 patchset (4960210 - 1.7GB)
  7. CPU Apr2007 patches for Identity Management and OID installs - some 12 MB
  8. Metadata Repository Creation Assistant V10.1.2.0.2 (400MB zipfile)
    I chose the Windows version; basically it does not matter, it executes against remote databases
  • Balance (http://www.inlab.de/balance.html)
Yes, I am going to do the load balancing as well - just take a look at either the enterprise setup or the high availability setup: both have loadbalancers. Also, these beasts have names. Now, I may have a few buck to spare for a gigabit switch, buying loadbalancing routers is a tad too much. So I will use non-dedicated software based (as opposed to firmware) load balancing. Should be interesting!

Installation
Once done downloading the software, and redistributing over all systems (see the previous post), I started installing.

Installation phase 1: the databases

Well, nothing much to tell about installing that base and patch level on Windows, but for some tricks:
  1. Install the baseline version of the software, do not create a database, or select a prebaked one.
  2. Install ultraSearch from the Companion CD.
  3. Patch software, twice (patches 5337014 and 5948242)
Only then fire up the Database Creation Assistant. Make sure you have sga_target, java_pool_size, processes and some other stuff increased, or you wil have to do it later when creating the Repository! Here's my init.ora with correct settings:
db_block_size=8192
db_file_multiblock_read_count=16

open_cursors=300

db_domain="home.local"
db_name=db1020

background_dump_dest=D:\oracle\admin\db1020\bdump
core_dump_dest=D:\oracle\admin\db1020\cdump
user_dump_dest=D:\oracle\admin\db1020\udump

control_files=("D:\oracle\oradata\db1020\control01.ctl", "D:\oracle\oradata\db1020\control02.ctl", "D:\oracle\oradata\db1020\control03.ctl")

job_queue_processes=10
compatible=10.2.0.3.0

processes=400

sga_target=600M

audit_file_dest=D:\oracle\admin\db1020\adump
remote_login_passwordfile=EXCLUSIVE
pga_aggregate_target=122683392
db_cache_size=144M

undo_management=AUTO
undo_tablespace=UNDOTBS1

aq_tm_processes=2
shared_pool_size=175M
java_pool_size=120M

Installation phase 2: create the Repository
Unpack the zip file, and install the Metadata Repository Creation Assistant. After that, just run the bloody thing - not much to tell here, apart from the strange behavior where 23 datafiles, totalling 1.4GB of diskspace gets written, deleted and written again. I chose to have the repository related files all in one location (hey - this is just a demo!), but separate them from the other database datafiles, by using a one level deeper subdirectory "rep".
If the checks on paramaters fail, alter them. This is a fairly easy install.

Installation phase 3: prepare for Replication - create the second instance
I used RMAN clone database for this. I need a (clean) backup anyway, so here we go. There are two stages: making the backup, and restoring the clone:

Phase 3a: backup.

Open a Command Line Interface (MS Windows: Run-> cmd, *ix: your favorite shell)
set your environment variables
RMAN target /
shutdown
startup mount
backup database;

Phase 3b: clone.
Create the (empty) directories for the clone (data/admin)
Copy init.ora and alter directory paths and instancename
Add newly created instance to tnsnames.ora and listener.ora.
Start listener.
MS Windows only: create the service:
Open a Command Line Interface (MS Windows: Run-> cmd)
oradim -new -sid oidrep -pfile D:\oracle\admin\oidrep\pfile\initoidrep.ora

Create passwordfile:

orapwd.exe file=%ORACLE_HOME%\database\PWDoidrep.ora password=oracle force=y
set oracle_sid=oidrep
sqlplus / as sysdba
startup nomount pfile=D:\oracle\admin\oidrep\pfile\initoidrep.ora
exit

Let's clone!
set oracle_sid=db1020
rman
connect target /
connect auxiliary sys/oracle@oidrep.home.local

duplicate target database to oidrep
pfile=D:\oracle\admin\oidrep\pfile\initoidrep.ora
db_file_name_convert=(
'D:\oracle\oradata\db1020', 'D:\oracle\oradata\oidrep',
'D:\oracle\oradata\db1020\rep', 'D:\oracle\oradata\oidrep\rep')
logfile 'D:\oracle\oradata\oidrep\redo01.log' size 100M,
'D:\oracle\oradata\oidrep\redo02.log' size 100M,
'D:\oracle\oradata\oidrep\redo03.log' size 100M;

That is it! This is a screen scrape from the actual session (some lines are snipped for brevity):
C:\Documents and Settings\frankbo>oradim -new -sid oidrep -pfile D:\oracle\admin\oidrep\pfile\initoidrep.ora
Instance created.

C:\Documents and Settings\frankbo>orapwd.exe file=%ORACLE_HOME%\database\PWDoidrep.ora password=oracle force=y

C:\Documents and Settings\frankbo>set oracle_sid=oidrep

C:\Documents and Settings\frankbo>sqlplus / as sysdba

SQL*Plus: Release 10.2.0.3.0 - Production on Sun Jun 3 13:38:53 2007

Copyright (c) 1982, 2006, Oracle.  All Rights Reserved.

Connected to an idle instance.

SQL> startup nomount pfile=D:\oracle\admin\oidrep\pfile\initoidrep.ora
ORACLE instance started.

Total System Global Area  629145600 bytes
Fixed Size                  1292132 bytes
Variable Size             318769308 bytes
Database Buffers          301989888 bytes
Redo Buffers                7094272 bytes
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Production
With the Partitioning, OLAP and Data Mining options
C:\Documents and Settings\frankbo>set oracle_sid=db1020
C:\Documents and Settings\frankbo>rman
Recovery Manager: Release 10.2.0.3.0 - Production on Sun Jun 3 13:41:49 2007
Copyright (c) 1982, 2005, Oracle.  All rights reserved.

RMAN> connect target /
connected to target database: DB1020 (DBID=4124432604)

RMAN> connect auxiliary sys/oracle@oidrep.home.local
connected to auxiliary database: OIDREP (not mounted)

RMAN> duplicate target database to oidrep
2> pfile=D:\oracle\admin\oidrep\pfile\initoidrep.ora
3> db_file_name_convert=(
4>  'D:\oracle\oradata\db1020', 'D:\oracle\oradata\oidrep',
5>  'D:\oracle\oradata\db1020\rep', 'D:\oracle\oradata\oidrep\rep')
6> logfile 'D:\oracle\oradata\oidrep\redo01.log' size 100M,
7>  'D:\oracle\oradata\oidrep\redo02.log' size 100M,
8>  'D:\oracle\oradata\oidrep\redo03.log' size 100M;

Starting Duplicate Db at 03-JUN-07
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: sid=432 devtype=DISK

contents of Memory Script:
{
set newname for datafile  1 to
"D:\ORACLE\ORADATA\OIDREP\SYSTEM01.DBF";
set newname for datafile  2 to
[snip - this goes on and on]
"D:\ORACLE\ORADATA\OIDREP\REP\GDEFAULT1_OID.DBF";
set newname for datafile  27 to
"D:\ORACLE\ORADATA\OIDREP\REP\SVRMG1_OID.DBF";
restore
check readonly
clone database
;
}
executing Memory Script

executing command: SET NEWNAME
[snipped more of the same]
executing command: SET NEWNAME

Starting restore at 03-JUN-07
using channel ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: starting datafile backupset restore
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
restoring datafile 00001 to D:\ORACLE\ORADATA\OIDREP\SYSTEM01.DBF
restoring datafile 00002 to D:\ORACLE\ORADATA\OIDREP\UNDOTBS01.DBF
[snip - this goes on and on]
restoring datafile 00027 to D:\ORACLE\ORADATA\OIDREP\REP\SVRMG1_OID.DBF
channel ORA_AUX_DISK_1: reading from backup piece D:\ORACLE\DB\10.2.0\DATABASE\01IIVGIE_1_1
channel ORA_AUX_DISK_1: restored backup piece 1
piece handle=D:\ORACLE\DB\10.2.0\DATABASE\01IIVGIE_1_1 tag=TAG20070529T215523
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:09:37
Finished restore at 03-JUN-07
sql statement: CREATE CONTROLFILE REUSE SET DATABASE "OIDREP" RESETLOGS NOARCHIVELOG
MAXLOGFILES     16
MAXLOGMEMBERS      3
MAXDATAFILES      100
MAXINSTANCES     8
MAXLOGHISTORY      292
LOGFILE
GROUP  1 'D:\oracle\oradata\oidrep\redo01.log' SIZE 100 M ,
GROUP  2 'D:\oracle\oradata\oidrep\redo02.log' SIZE 100 M ,
GROUP  3 'D:\oracle\oradata\oidrep\redo03.log' SIZE 100 M
DATAFILE
'D:\ORACLE\ORADATA\OIDREP\SYSTEM01.DBF'
CHARACTER SET WE8MSWIN1252


contents of Memory Script:
{
switch clone datafile all;
}
executing Memory Script

released channel: ORA_AUX_DISK_1
datafile 2 switched to datafile copy
input datafile copy recid=1 stamp=624289967 filename=D:\ORACLE\ORADATA\OIDREP\UNDOTBS01.DBF
datafile 3 switched to datafile copy
input datafile copy recid=2 stamp=624289967 filename=D:\ORACLE\ORADATA\OIDREP\SYSAUX01.DBF
[snip - this goes on and on]
datafile 27 switched to datafile copy
input datafile copy recid=26 stamp=624289971 filename=D:\ORACLE\ORADATA\OIDREP\REP\SVRMG1_OID.DBF

contents of Memory Script:
{
recover
clone database
noredo
,
delete archivelog
;
}
executing Memory Script

Starting recover at 03-JUN-07
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: sid=431 devtype=DISK
Finished recover at 03-JUN-07

contents of Memory Script:
{
shutdown clone;
startup clone nomount pfile= 'D:\oracle\admin\oidrep\pfile\initoidrep.ora';
}
executing Memory Script

database dismounted
Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area     629145600 bytes

Fixed Size                     1292132 bytes
Variable Size                318769308 bytes
Database Buffers             301989888 bytes
Redo Buffers                   7094272 bytes
sql statement: CREATE CONTROLFILE REUSE SET DATABASE "OIDREP" RESETLOGS NOARCHIVELOG
MAXLOGFILES     16
MAXLOGMEMBERS      3
MAXDATAFILES      100
MAXINSTANCES     8
MAXLOGHISTORY      292
LOGFILE
GROUP  1 'D:\oracle\oradata\oidrep\redo01.log' SIZE 100 M ,
GROUP  2 'D:\oracle\oradata\oidrep\redo02.log' SIZE 100 M ,
GROUP  3 'D:\oracle\oradata\oidrep\redo03.log' SIZE 100 M
DATAFILE
'D:\ORACLE\ORADATA\OIDREP\SYSTEM01.DBF'
CHARACTER SET WE8MSWIN1252


contents of Memory Script:
{
set newname for tempfile  1 to
"D:\ORACLE\ORADATA\OIDREP\TEMP01.DBF";
switch clone tempfile all;
catalog clone datafilecopy  "D:\ORACLE\ORADATA\OIDREP\UNDOTBS01.DBF";
catalog clone datafilecopy  "D:\ORACLE\ORADATA\OIDREP\SYSAUX01.DBF";
[snip - this goes on and on]
catalog clone datafilecopy  "D:\ORACLE\ORADATA\OIDREP\REP\GDEFAULT1_OID.DBF";
catalog clone datafilecopy  "D:\ORACLE\ORADATA\OIDREP\REP\SVRMG1_OID.DBF";
switch clone datafile all;
}
executing Memory Script

executing command: SET NEWNAME

renamed temporary file 1 to D:\ORACLE\ORADATA\OIDREP\TEMP01.DBF in control file

cataloged datafile copy
datafile copy filename=D:\ORACLE\ORADATA\OIDREP\UNDOTBS01.DBF recid=1 stamp=624289989

cataloged datafile copy
datafile copy filename=D:\ORACLE\ORADATA\OIDREP\SYSAUX01.DBF recid=2 stamp=624289989

cataloged datafile copy
datafile copy filename=D:\ORACLE\ORADATA\OIDREP\USERS01.DBF recid=3 stamp=624289989
[snip - this goes on and on]
cataloged datafile copy
datafile copy filename=D:\ORACLE\ORADATA\OIDREP\REP\SVRMG1_OID.DBF recid=26 stamp=624289994

datafile 2 switched to datafile copy
input datafile copy recid=1 stamp=624289989 filename=D:\ORACLE\ORADATA\OIDREP\UNDOTBS01.DBF
datafile 3 switched to datafile copy
input datafile copy recid=2 stamp=624289989 filename=D:\ORACLE\ORADATA\OIDREP\SYSAUX01.DBF
datafile 4 switched to datafile copy
[snip - this goes on and on]
datafile 27 switched to datafile copy
input datafile copy recid=26 stamp=624289994 filename=D:\ORACLE\ORADATA\OIDREP\REP\SVRMG1_OID.DBF

contents of Memory Script:
{
Alter clone database open resetlogs;
}
executing Memory Script

database opened
Finished Duplicate Db at 03-JUN-07

RMAN> exit
Recovery Manager complete.

C:\Documents and Settings\frankbo>time /t
01:54 PM

So - all in all the cloning took 13 minutes.
In contrast to earlier releases, that did not create the tempfile, belonging to the temporary tablespace, there is no more need to create a tempfile - it's there!
C:\Documents and Settings\frankbo>set oracle_sid=oidrep
C:\Documents and Settings\frankbo>sqlplus / as sysdba

SQL*Plus: Release 10.2.0.3.0 - Production on Sun Jun 3 14:02:11 2007
Copyright (c) 1982, 2006, Oracle.  All Rights Reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Production
With the Partitioning, OLAP and Data Mining options

SQL> select name from v$tempfile;
NAME
--------------------------------------------------------------------------------
D:\ORACLE\ORADATA\OIDREP\TEMP01.DBF

SQL> create spfile from pfile='D:\oracle\admin\oidrep\pfile\initoidrep.ora';
File created.

Bounce the db, to make sure the spfile is picked up
SQL> show parameter pfile

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      D:\ORACLE\DB\10.2.0\DATABASE\SPFILEOIDREP.ORA
SQL> select dbid, db_unique_name from v$database;

DBID       DB_UNIQUE_NAME
---------- ------------------------------
3574270531 oidrep

SQL> connect sys/manager@db1020.home.local as sysdba
Connected.
SQL> /

DBID DB_UNIQUE_NAME
---------- ------------------------------
4124432604 db1020

No worries about dbid, either...

That concludes the database preparations.

Installation phase 4: prepare the Network
I made a distinction between two stages: getting the balancer, and adding virtual addresses.
First off, a little bit about the setup. As said earlier on, I (only) have 3 machines, and the complete configuration requires ate least four, better yet, six. As 6=3*2, every machine gets a double function, some even triple functions (and no - you do not want to VMWare this - your host will not cope with it...)

Phase 4a: get, make and install balance.
Logon to your machines as root. I need the c-compiler, so let's get it:
yum install gcc

Next, download the source tarball from http://www.inlab.de/balance-3.35.tar.gz to /install
[root@idmhost ~]# cd /install/
[root@idmhost install]# gunzip balance-3.35.tar.gz
[root@idmhost install]# tar -xf balance-3.35.tar
[root@idmhost install]# cd balance-3.35

Now, if you would run make make install at this stage, you would get a (minor) error; there's a slight typographical error on line 11 of the makefile, so change the Makefile file:
#MANDIR=${BINDIR}/../man/man1
MANDIR=/usr/share/man/man1

And run "make install":
[root@idmhost balance-3.35]# make install
install -o root -g root -m 755  balance \
/usr/sbin/balance
install -o root -g root -m 755  balance.1 \
/usr/share/man/man1
mkdir -p /var/run/balance
chmod 1777 /var/run/balance
[root@idmhost balance-3.35]#

Not doing so, will lead to this error:
[root@idmhost balance-3.35]# make install
install -o root -g root -m 755  balance \
/usr/sbin/balance
install -o root -g root -m 755  balance.1 \
/usr/sbin/../man/man1
install: cannot create regular file `/usr/sbin/../man/man1': No such file or directory
make: *** [install] Error 1
[root@idmhost balance-3.35]#

No harm done, simply edit the Makefile, and rerun... Failure to do so will not have any effect on the program, you just will not have the man-pages.

Phase 4b: virtual addresses and names to your local network.
Remember, basically, I wanted:
  • A load-balanced request to two SSO servers.
  • Those SSO servers request a loadbalanced OID.
  • Those two OID processes use SQL*Net time out and loadbalancing to query two active databases, which are clones of eachother.
Remember also, I only have three machines.

So, network wise, I would need:
  • Two SSO instances (addresses: IDM_IP1 and IDM_IP2), being served by a loadbalancer. This loadbalancer is actually a HTTP balancer, serving the SSO and DAS pages (the Identity Management Layer).
    As you do not want to bother people with the distinction between IDM_IP1 or IDM_IP2, the balancer should have a name. From now on, that is login.home.local. Ip address is IDM_IP0.
  • The SSO/DAS pages are requesing OID services through a load balancer, but that is an LDAP loadbalancer. It only needs to server LDAP requests (I am going to use the non-priveleged port range, 3060 (non-ssl) and 3130 (SSL), in stead of the default 386 and 636).
Not sure if that balancer should have a name; it's address ip OID_OP0, and it balances between OID_IP1 and OID_IP2.

Physically, I have used the machine names OIDHOST and IDMHOST so far. See previous posting about that. What I'm going to do, is install the first OID and IDM installs on the OIDHOST and IDMHOST repectively, and the second OID and IDM installs go on the IDMHOST and OIDHOST respectively.
Both application servers will serve OID as well as IDM:
IDMHOST:    IDM1    OID2
OIDHOST:    IDM2    OID1
The first loadbalancer, the HTTP one, will sit on IDMHOST, the second will sit on OIDHOST.
This leads to:
IDMHOST/original address:   192.168.1.220
IDMHOST/IDM1 address:       192.168.1.225
IDMHOST/OID2 address:       192.168.1.226
IDMHOST/login.home.local:   192.168.1.227
OIDHOST/original address:   192.168.1.210
OIDHOST/IDM2 address:       192.168.1.215
OIDHOST/OID1 address:       192.168.1.216
OIDHOST/ldapbal.home.local: 192.168.1.217

In addition: db1020.home.local resides on 192.128.1.104, as does oidrep, the replication instance.

Let's add the addresses:
On IDMHOST:
ifconfig eth0:1 192.168.1.225
ifconfig eth0:2 192.168.1.226
ifconfig eth0:3 192.168.1.227

On OIDHOST:
ifconfig eth0:1 192.168.1.215
ifconfig eth0:2 192.168.1.216
ifconfig eth0:3 192.168.1.217

Check by running ifconfig:
eth0      Link encap:Ethernet  HWaddr 00:50:DA:4A:BC:2A
inet addr:192.168.1.210  Bcast:192.168.1.255  Mask:255.255.255.0
inet6 addr: fe80::250:daff:fe4a:bc2a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:12641 errors:0 dropped:0 overruns:1 frame:0
TX packets:9048 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:9696620 (9.2 MiB)  TX bytes:1106121 (1.0 MiB)
Interrupt:169 Base address:0xd800

eth0:1    Link encap:Ethernet  HWaddr 00:50:DA:4A:BC:2A
inet addr:192.168.1.215  Bcast:192.168.1.255  Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
Interrupt:169 Base address:0xd800

eth0:2    Link encap:Ethernet  HWaddr 00:50:DA:4A:BC:2A
inet addr:192.168.1.216  Bcast:192.168.1.255  Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
Interrupt:169 Base address:0xd800

eth0:3    Link encap:Ethernet  HWaddr 00:50:DA:4A:BC:2A
inet addr:192.168.1.217  Bcast:192.168.1.255  Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
Interrupt:169 Base address:0xd800
Or use Webmin, the Networking entry, Network Configuration, Network Interfaces.
You may alse define (permanent) virtual addresses here. If you insist on doing it by hand, create the appropiate files (ifcfg-eth0:1, etc) in /etc/sysconfig/network-scripts:
BOOTPROTO=none
DEVICE=eth0:1
NETMASK=255.255.255.0
MTU=1500
BROADCAST=192.168.1.255
ONPARENT=yes
IPADDR=192.168.1.225
NETWORK=192.168.1.0
ONBOOT=yes

Alternatively, add the ifconfig eth0:1 lines to /etc/rc.local:

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local
/sbin/ifconfig eth0:1 192.168.1.215
/sbin/ifconfig eth0:2 192.168.1.216
/sbin/ifconfig eth0:3 192.168.1.217

Change the hosts files on all machines, under Linux, it is /etc/hosts:
127.0.0.1               localhost.localdomain localhost
192.168.1.210           oidhost.home.local
192.168.1.220           idmhost.home.local
192.168.1.104           dbhost.home.local
192.168.1.225           idm1.home.local idm1
192.168.1.226           oid2.home.local oid2
192.168.1.227           login.home.local login
192.168.1.215           idm2.home.local idm2
192.168.1.216           oid1.home.local oid1
192.168.1.217           ldapbalancer.home.local ldapbalancer

Do not foget to add these to the database host (C:\WINDOWS\system32\drivers\etc)! Failing in doing so will reslove in nasty install errors
(ORA-31203: DBMS_LDAP: PL/SQL - Init Failed, java class not found)

Let's reboot the systems to see if everything acts as we want: reboot -n
Try to ping every host defined, from every machine. If that is successfull, let's do the vandango:
[root@idmhost ~]# balance -b login.home.local http idm1:http % idm2:http %
[root@idmhost ~]# balance -b login.home.local https idm1:https % idm2:https %

Similar:
[root@oidhost ~]# balance -b ldapbalancer.home.local 3060 oid1:3060 oid2:3060
[root@oidhost ~]# balance -b ldapbalancer.home.local 3130 oid1:3130 oid2:3130

That concludes phase 4.
Installation phase 5: Oracle Internet DirectoryPhase 5a: Preliminaries.
On both machines, create distinct groups and user:

[root@oidhost ~]# groupadd oidown
[root@oidhost ~]# groupadd oidinst
[root@oidhost ~]# useradd oidoracle -g oidinst -G oidown -c 'Oracle Internet Directory software owner'
[root@oidhost ~]# passwd oidoracle
Changing password for user oidoracle.
New UNIX password:
Retype new UNIX password:
passwd: all authentication tokens updated successfully.

Make sure I can unpack the cpio and zipped files in the /install directory (which is not owned by oidoracle!)

[root@oidhost ~]# chmod 777 /install

Create the installation directory, and change ownership:

[root@oidhost ~]# mkdir -p /oracle/ias/oraInventory
[root@oidhost ~]# chown -R oidoracle:oidown /oracle
[root@oidhost ~]# su - oidoracle
[oidoracle@oidhost ~]$ cd /install
[oidoracle@oidhost install]$ cpio -idmv< /install/as_linux_x86_portal_wireless_101202_disk1.cpio [oidoracle@oidhost install]$ cpio -idmv< /install/as_linux_x86_portal_wireless_101202_disk2.cpio [oidoracle@oidhost install]$ cpio -idmv< /install/as_linux_x86_portal_wireless_101202_disk3.cpio [oidoracle@oidhost install]$ cpio -idmv< /install/as_linux_x86_portal_wireless_101202_disk4.cpio [oidoracle@oidhost install]$ unzip p4960210_10122_LINUX.zip -d p4960210 [oidoracle@oidhost install]$ unzip p5901894_10122_LINUX.zip -d p5901894 [oidoracle@oidhost install]$ unzip p5922121_10122_LINUX.zip -d p5922121

Phase 5a: first OID install.
I was planning on using non-default ports, so let's do some prepartion for that:
[oidoracle@oidhost install]$ cp Disk1/stage/Response/staticports.ini /oracle/ias/staticports.ini
Now, I need to make the installer aware of the fact, I want ports 3060 and 3130 to be used. The interesting part of staticports.ini:
# Infrastructure

Oracle Internet Directory port = 3060
Oracle Internet Directory (SSL) port = 3130
#Oracle Certificate Authority SSL Server Authentication port = port_num
#Oracle Certificate Authority SSL Mutual Authentication port = port_num
#Ultra Search HTTP port number = port_num

OK - let's fire up the installer:
[oidoracle@oidhost ~]$ export DISPLAY=192.168.1.104:0.0
[oidoracle@oidhost ~]$ /install/Disk1/runInstaller

Enter the correct locations, and...

Let's do what is asked for...

Once more - correct locations...

Infrastucture install...


Let's do Identity Management.


Yeah - don't feel like upping it even further - besides, these are the values, specified in the Linux Installation Manual... Just mark them as okay, and continue.
Of course we have root priveleges - I am not really going to upload a picture showing how to confirm that, just continue to the next:


Remember the envisioned setup: The LDAP services (OID) and Integration will be running here, and the rest (SSO and DAS, no CA this time) on the Identity Management Host (idmhost.home.local)

With all the preparations, make sure we use them! Select the correct file.



Ebter the correct data, and...

What the ...?!? The Oracle Application Server Metadata Repository is not compatible?!? I checked and doublechecked versions - no error there! Back to the drawingboard!

Update:
As far as I can tell, Metalink came up empty, Google came up empty and so did tahiti. I admit, I did not look at all references matching my search criteria, because a lot of hits are about backwards compatibility problems. And I know for a fact, the MCRA versions 10.1.2.0.0 and 10.1.2.0.2 are incompatble, too.
The screen itself leaves no room for informative queries, so all that is left is the log file of the installation itself. This looks like:
Calling Query DBConnectQueries8.2  GetSchemaVer

SchemaName = *Protected value, not to be logged*

SchemaPassword = *Protected value, not to be logged*

ConnectString = 192.168.1.104:1521/db1020.home.local

SqlQuery = select attrval from ods.ds_attrstore where entryid=1 and attrname = 'orcldirectoryversion'
Query Returned: OID 10.1.2.1.0
OID Schema value returned from SQL is OID 10.1.2.1.0.

Extracted version is 10.1.2.1.0.
Calling Query DBConnectQueries8.2  IsOIDConfigured

SchemaName = *Protected value, not to be logged*

SchemaPassword = *Protected value, not to be logged*

ConnectString = 192.168.1.104:1521/db1020.home.local
Query Returned: false
Calling Query DBConnectQueries8.2  IsUserWithDBAPriv

User = *Protected value, not to be logged*

Password = *Protected value, not to be logged*

ConnectString = 192.168.1.104:1521/db1020.home.local
Query Returned: true
Calling Query DBConnectQueries8.2  GetRepositoryVer

User = *Protected value, not to be logged*

Password = *Protected value, not to be logged*

ConnectString = 192.168.1.104:1521/db1020.home.local
Query Returned: Null
Using the default value for query.
Error:*** Alert: The Oracle Application Server Metadata Repository that you have specified is not a compatible version for configuring Oracle Internet Directory. Please specify another database. ***

What I understand from this, is the fact that OID is not configured, causes the installer to abort. Of course OID isn't configured - I choose to install that!
Anyway - somewhere deep (in /install/Disk1/stage/Queries/DBConnectQueries/8.2/1) there is a file, called DBConnectQueries.jar. Opening it, and searching for GetRepositoryVer showed some interesting stuff (like the development machine, syndey.oracle.com, with system password!), like:
select version from app_registry where comp_id = 'MRC';
select version from ias_versions where id = 'mrc';

I cannot tell where the second query comes in, but the first does resolve:
SQL> select comp_id, version, status from app_registry;

COMP_ID                        VERSION                        STATUS
------------------------------ ------------------------------ -----------
PORTAL                         10.1.2.0.2                     VALID
SSO                            10.1.2.0.2                     VALID
WORKFLOW                       10.1.2.0.2                     VALID
B2B                            10.1.2.0.2                     VALID
BAM                            10.1.2.0.2                     VALID
MRC                                                           LOADING
OCA                            10.1.2.0.2                     VALID
OID                            10.1.2.0.2                     VALID
DCM                            10.1.2.0.2                     VALID
DISCOVERER                     10.1.2.0.2                     VALID


I fired up the MRCA again, and tried to redo the install. Nope - remove first, and only then install... Remove drops objects, before dropping tablespaces. There is a faster way to do that... had to do it twice, no indication why, the last line of the first sessions' log reads:
Repository Loader actionStarting
The correct, completed session goes on after that:
Repository Loader actionStarting
Repository Loader actionFinished
Repository Loader ActionQueueFinished
Unloading...
And continues dropping tablespaces, and explaining the wizard has stopped, about twenty times. Mysteries...

During the process, I observed:
SQL> select comp_id, version, status from app_registry;
no rows selected

SQL> /

COMP_ID                        VERSION                        STATUS
------------------------------ ------------------------------ -----------
MRC                                                           LOADING
DISCOVERER                     10.1.2.0.2                     VALID
DCM                            10.1.2.0.2                     VALID

SQL> /

COMP_ID                        VERSION                        STATUS
------------------------------ ------------------------------ -----------
PORTAL                                                        LOADING
SSO                            10.1.2.0.2                     VALID
WORKFLOW                       10.1.2.0.2                     VALID
B2B                            10.1.2.0.2                     VALID
BAM                            10.1.2.0.2                     VALID
MRC                                                           LOADING
OCA                            10.1.2.0.2                     VALID
OID                            10.1.2.0.2                     VALID
DISCOVERER                     10.1.2.0.2                     VALID
DCM                            10.1.2.0.2                     VALID

10 rows selected.
SQL> /

COMP_ID                        VERSION                        STATUS
------------------------------ ------------------------------ -------
SYNDICATION                    10.1.2.0.2                     VALID
PORTAL                         10.1.2.0.2                     VALID
SSO                            10.1.2.0.2                     VALID
WORKFLOW                       10.1.2.0.2                     VALID
B2B                            10.1.2.0.2                     VALID
BAM                            10.1.2.0.2                     VALID
MRC                            10.1.2.0.2                     VALID
OCA                            10.1.2.0.2                     VALID
OID                            10.1.2.0.2                     VALID
WIRELESS                       10.1.2.0.2                     VALID
DISCOVERER                     10.1.2.0.2                     VALID
DCM                            10.1.2.0.2                     VALID
WCS                            10.1.2.0.2                     VALID
UDDI                           10.1.2.0.2                     VALID


That seems to be different from where I started - but the MCRA did finish OK...
Well, back to cloning and then retry the install!

Update:
Started the machines, database instance and listener, balancer om both machines.
Checked hosts. Installer continued smoothly this time:

I left it for what it was - you may consider otherwise, especially when you have plans on extending the root entry (.local, in this case). For .com it may not be such a problem, but for .nl it will be - imagine your company extends abroad. In that case, consider a megalomaniac '.world' as root: your.company.nl.world can expand into your.other.be.world.

MDS stands for Master Definition Site...

229 products(!) to be installed. And I did not even select all options!

Let's take a closer look at the log, then:
Leaving Ldap Post Installation Set File Permissions
Stopping  OID Server using OPMN..
Starting OID Server using OPMN..
Mon Jun 18 19:09:49 CEST 2007 Bind request issued. Waiting for OID Server response.
with a retryCount:20
Mon Jun 18 19:10:19 CEST 2007 Bind request issued. Waiting for OID Server response.
javax.naming.CommunicationException: oidhost.home.local:3060 [Root exception is java.net.ConnectException: Connection refused]

OK - see if the process actually runs; switch to $ORACLE_HOME/opmn/bin, and:
[oidoracle@oidhost bin]$ ./opmnctl status

Processes in Instance: mds.oidhost.home.local
-------------------+--------------------+---------+---------
ias-component      | process-type       |     pid | status
-------------------+--------------------+---------+---------
DSA                | DSA                |     N/A | Down
LogLoader          | logloaderd         |     N/A | Down
dcm-daemon         | dcm-daemon         |    5816 | Alive
HTTP_Server        | HTTP_Server        |     N/A | Down
OID                | OID                |     N/A | Down
No wonder, OID is down... Let's just start all processes:
[oidoracle@oidhost bin]$ ./opmnctl startall
opmnctl: starting opmn and all managed processes...
[oidoracle@oidhost bin]$ ./opmnctl status

Processes in Instance: mds.oidhost.home.local
-------------------+--------------------+---------+---------
ias-component      | process-type       |     pid | status
-------------------+--------------------+---------+---------
DSA                | DSA                |     N/A | Down
LogLoader          | logloaderd         |     N/A | Down
dcm-daemon         | dcm-daemon         |    5816 | Alive
HTTP_Server        | HTTP_Server        |    7753 | Alive
OID                | OID                |    7758 | Alive

Still - retry fails. Then I realize, I already switched on loadbalancing... and sure enough, after killing these balance processes, the wizards continued, only to fail once more:

This is a bit of a silly error message: opmn cannot start the process, because I already started it! Resolution: stop the process manually:
[oidoracle@oidhost bin]$ ./opmnctl stopproc type=ohs
opmnctl: stopping opmn managed processes...

Some (actually, a lot) of wizards later, this is the reward:


Update: (Phase 5c-second OID install)Started both instances, and opened the databases. Logged on to oidhost, and changed .bash_profile; added those lines:
export ORACLE_HOME=/oracle/ias/oid10.1.2
export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/opmn/bin:$PATH

That allows me to:
[oidoracle@oidhost ~]$ opmnctl startall
opmnctl: starting opmn and all managed processes...
[oidoracle@oidhost ~]$ opmnctl status

Processes in Instance: mds.oidhost.home.local
-------------------+--------------------+---------+---------
ias-component      | process-type       |     pid | status
-------------------+--------------------+---------+---------
DSA                | DSA                |     N/A | Down
LogLoader          | logloaderd         |     N/A | Down
dcm-daemon         | dcm-daemon         |    3906 | Init
HTTP_Server        | HTTP_Server        |    3904 | Alive
OID                | OID                |    3912 | Alive

Logged on to the idmhost, with oidoracle account. Editied the localhosts file again, with the following contents:
Oracle Internet Directory port = 3060
Oracle Internet Directory (SSL) port = 3130
#Oracle Certificate Authority SSL Server Authentication port = port_num
#Oracle Certificate Authority SSL Mutual Authentication port = port_num
#Ultra Search HTTP port number = port_num

Fired up the installer:
[oidoracle@idmhost oracle]$ /install/Disk1/runInstaller -paramFile /oracle/ias/oraparam.ini

Only screens that do differ from above are loaded:
Select three options: Internet Directory, Directory Integration and HA/Replication.


Indicate the correct location of the staticports.ini file.
I had to use SYSTEM here - could not get SYS to work:

Hmmmmmm.... I don't want to choose here! I want both. Maybe this is the reason clustered installs don't replicate? In this manner, there are two farms, and farms cannot cluster. Only whatever application server instance belongs to the farm, can participate in a cluster: 1 farm == 1 repository.
Maybe when I base the instance on a file-based repository, on a shared disk?!?

Next screen, select Replication:


Next screen, select Advanced Replication.


Now, this one is tricky: it states "Master Node", where in fact, this is the second install. True, but this is Multi Master Replication, so in fact: there are no masters (or everyone is the master)!


Same here: "Master", but watch out: the data entered actually refers to the real master, the first installed instance: oidhost.home.local!


Provide the correct connection information, and get used to the "cn=" notation - this is LDAP land... Note the naming of the instance: rms, as in "Replicated Master Site".


That's it... the installer will install, the wizzards wizz, and it all ends in:


Update: Something went wrong, I noticed after reflection. I miss one installer screen; the one that allows me to select the (virtual) ip address and (virtual) server name! It should have been presented because of the changes I made to oraparam.ini (SHOW_HOSTNAME=ALWAYS_SHOW) .

Update:
Before attempting to get replication to work, I'll need to fix the network component. That means adding the "other" entry to each tnsnames.ora, so each file is identical:
OIDREP.home.local =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = dbhost.home.local)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = oidrep.home.local)
)
)

db1020.home.local =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = dbhost.home.local)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = db1020.home.local)
)

It also means, I need to add a default domain - OID seems to make it a habit of sometimes using a domain qualified call, sometimes not. Consequently, db1020 as well as db1020.home.local must be resolved. Added this to sqlnet.ora:
names.default_domain=home.local

The same is true for the database server(s); they need to be able to connect lateron - afterall, it is database based replication, not Application Server!

Next stop: replication!

Update: (Phase 6 - install Replication)
After all these preparations, starting replication should be quite easy: use the remtool (reminding me of a REMoval tool, what's in a name?): (some logging has been snipped to save space)
[oidoracle@oidhost oid10.1.2]$ remtool -asrsetup -v
------------------------------------------------------------------------------
ASR Setup for OID Replication
WARNING:
Make sure that the replication administrator that you
enter below does not exist already in any of the nodes
that will be part of the DRG to be created now. If the
user exists, that user will be dropped and will be
created newly.
------------------------------------------------------------------------------
Enter replication administrator's name       : repadmin

Enter replication administrator's password   :
Reenter replication administrator's password :
Enter Master Definition Site (MDS) details   :
Enter global name of MDS                     : db1020.home.local

Enter SYSTEM user password of MDS            :
Enter Remote Master Site (RMS) details       :
Enter global name of RMS #  1                : oidrep.home.local

Enter SYSTEM user password of RMS #  1       :
Are there more Remote Master Sites in the group? [y/n/q] : n

Verify the details you had entered.
------------------------------------------------------------------------------
Replication administrator's name   : repadmin
Master Definition Site             : db1020.home.local
Remote Master Site #  1            : oidrep.home.local
Are these details correct? [y/n/q] : y

------------------------------------------------------------------------------
ASR setup in progress...

DB1020.HOME.LOCAL : Verifying uniqueness of replication agreement entry...
DB1020.HOME.LOCAL : Dropping replication administrator repadmin...
DB1020.HOME.LOCAL : Creating replication administrator repadmin...
DB1020.HOME.LOCAL : Granting privileges or roles required for replication administrator to repadmin...
DB1020.HOME.LOCAL : Granting privileges or roles required for replication administrator to repadmin...
DB1020.HOME.LOCAL : Granting privileges or roles required for replication administrator to repadmin...
DB1020.HOME.LOCAL : Creating purge job...
DB1020.HOME.LOCAL : Dropping database link made to OIDREP.HOME.LOCAL...
DB1020.HOME.LOCAL : Dropping database link made to OIDREP.HOME.LOCAL...
DB1020.HOME.LOCAL : Creating database link to OIDREP.HOME.LOCAL...
DB1020.HOME.LOCAL : Scheduling push job to OIDREP.HOME.LOCAL...
OIDREP.HOME.LOCAL : Verifying uniqueness of replication agreement entry...
OIDREP.HOME.LOCAL : Dropping replication administrator repadmin...
OIDREP.HOME.LOCAL : Creating replication administrator repadmin...
OIDREP.HOME.LOCAL : Granting privileges or roles required for replication administrator to repadmin...
OIDREP.HOME.LOCAL : Granting privileges or roles required for replication administrator to repadmin...
OIDREP.HOME.LOCAL : Granting privileges or roles required for replication administrator to                       repadmin...
OIDREP.HOME.LOCAL : Creating purge job...
OIDREP.HOME.LOCAL : Dropping database link made to DB1020.HOME.LOCAL...
OIDREP.HOME.LOCAL : Creating database link to DB1020.HOME.LOCAL...
OIDREP.HOME.LOCAL : Scheduling push job to DB1020.HOME.LOCAL...
DB1020.HOME.LOCAL : Dropping replication group LDAP_REP...
DB1020.HOME.LOCAL : Creating replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ODS.ASR_CHG_LOG to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ODS.ODS_CHG_STAT to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSSO_LS_CONFIGURATION_INFO_T to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSSO_PS_CONFIGURATION_INFO_T to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSSO_PAPP_CONFIGURATION_INF_T to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSSO_PSEX_APP_INFO$ to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSSO_PSEX_USER_INFO$ to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSSO_ANNOUNCEMENT_CONFIG_T to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWHOSTING_SWITCH$ to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSEC_PERSON$ to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWCTX_COOKIE_INFO$ to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSSO_APPLICATION_INFO_T to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSSO_APPUSERINFO_T to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSEC_ENABLER_CONFIG_INFO$ to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSUB_MODEL$ to replication group LDAP_REP...
OIDREP.HOME.LOCAL : Dropping replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding replication site OIDREP.HOME.LOCAL to replication group LDAP_REP...
DB1020.HOME.LOCAL : Executing deferred administrative requests...
OIDREP.HOME.LOCAL : Executing deferred administrative requests...
DB1020.HOME.LOCAL : Generating replication support for TABLE ODS.ASR_CHG_LOG...
DB1020.HOME.LOCAL : Executing deferred administrative requests...
OIDREP.HOME.LOCAL : Executing deferred administrative requests...
DB1020.HOME.LOCAL : Generating replication support for TABLE ODS.ODS_CHG_STAT...
DB1020.HOME.LOCAL : Executing deferred administrative requests...
OIDREP.HOME.LOCAL : Executing deferred administrative requests...
DB1020.HOME.LOCAL : Generating replication support for TABLE ORASSO.WWSSO_LS_CONFIGURATION_INFO_T...
ORASSO.WWSSO_PS_CONFIGURATION_INFO_T...
ORASSO.WWSSO_PAPP_CONFIGURATION_INF_T...
ORASSO.WWSSO_PSEX_APP_INFO$...
ORASSO.WWSSO_PSEX_USER_INFO$...
ORASSO.WWSSO_ANNOUNCEMENT_CONFIG_T...
ORASSO.WWHOSTING_SWITCH$...
ORASSO.WWSEC_PERSON$...
ORASSO.WWCTX_COOKIE_INFO$...
ORASSO.WWSSO_APPLICATION_INFO_T...
ORASSO.WWSSO_APPUSERINFO_T...
ORASSO.WWSEC_ENABLER_CONFIG_INFO$...
ORASSO.WWSUB_MODEL$...
DB1020.HOME.LOCAL : Verifying initialization parameter...
DB1020.HOME.LOCAL : Altering init param value of global_names to TRUE...
CORRECTED:
DB1020.HOME.LOCAL : Initialization parameter global_names' value has been altered to TRUE.                       Alter INIT.ORA file to reflect the above change.
OIDREP.HOME.LOCAL : Verifying initialization parameter...
OIDREP.HOME.LOCAL : Altering init param value of global_names to TRUE...
CORRECTED:
OIDREP.HOME.LOCAL : Initialization parameter global_names' value has been altered to TRUE.                       Alter INIT.ORA file to reflect the above change.
DB1020.HOME.LOCAL : Verifying uniqueness of replication agreement entry...
OIDREP.HOME.LOCAL : Verifying uniqueness of replication agreement entry...
DB1020.HOME.LOCAL : Verifying replication agreement entry...
DB1020.HOME.LOCAL : Inserting replication agreement entry oidhost_db1020...
CORRECTED:
DB1020.HOME.LOCAL : "oidhost_db1020" hostname has been added to replication agreement entry.
DB1020.HOME.LOCAL : Inserting replication agreement entry idmhost_oidrep...
CORRECTED:
DB1020.HOME.LOCAL : "idmhost_oidrep" hostname has been added to replication agreement entry.
OIDREP.HOME.LOCAL : Verifying replication agreement entry...
OIDREP.HOME.LOCAL : Inserting replication agreement entry oidhost_db1020...
CORRECTED:
OIDREP.HOME.LOCAL : "oidhost_db1020" hostname has been added to replication agreement entry.
OIDREP.HOME.LOCAL : Inserting replication agreement entry idmhost_oidrep...
CORRECTED:
OIDREP.HOME.LOCAL : "idmhost_oidrep" hostname has been added to replication agreement entry.
DB1020.HOME.LOCAL : Resuming replication activity...
DB1020.HOME.LOCAL : Executing deferred administrative requests...
OIDREP.HOME.LOCAL : Executing deferred administrative requests...
------------------------------------------------------------------------------
ASR setup has been configured successfully.
------------------------------------------------------------------------------
Directory Replication Group (DRG) details :

-------- ------------- ----------------------- ------------- ------------- ----
Instance Host Name     Global Name             Version       Replicaid     Site
Name                                                                       Type
-------- ------------- ----------------------- ------------- ------------- ----
db1020   CS-FRANK03    DB1020.HOME.LOCAL       OID 10.1.2.1. oidhost_db102 MDS
oidrep   CS-FRANK03    OIDREP.HOME.LOCAL       OID 10.1.2.1. idmhost_oidre RMS
-------- ------------- ----------------------- ------------- ------------- ----

[oidoracle@oidhost oid10.1.2]$


If the setup fails with
ORA-12154: TNS:could not resolve the connect identifier specified
in the dropping/creating database links part, right at the beginning, make sure global_name (select * from global_name) is the same as your service_name in tnsnames.ora.

Now, start replication services, and see if they run:
[oidoracle@oidhost oid10.1.2]$ oidctl connect=db1020.home.local server=oidrepld instance=1 flags="-h oidhost.home.local -p 3060" start
[oidoracle@oidhost oid10.1.2]$ $ORACLE_HOME/ldap/bin/ldapcheck

Checking Oracle Internet Directory Processes ...ALL

Process oidmon is Alive as PID 3897
Process oidldapd is Alive as PID 3898
Process oidldapd is Alive as PID 3904
Process oidrepld is Alive as PID 8451
Process odisrv is Alive as PID 3899

Same thing on other machine:
[oidoracle@idmhost bin]$ oidctl connect=oidrep.home.local server=oidrepld instance=1 flags="-h idmhost.home.local -p 389" start
Waiting for OIDMON to stop OIDREPLD, see oidmon.log for details.
[oidoracle@idmhost bin]$ ./ldapcheck

Checking Oracle Internet Directory Processes ...ALL

Process oidmon is Alive as PID 3602
Process oidldapd is Alive as PID 3611
Process oidldapd is Alive as PID 3615
Process oidrepld is Alive as PID 5457
Process odisrv is Alive as PID 3612
Does it work?
Well, fire up the Directory Manager, connect to both LDAP servers, and navigate to cn=Entry Management,dc=local,dc=home,cn=users,cn=orcladmin.
On the first machine, oidhost, you will see this (notice the timestamp):


The replicated machine, idmhost, will show this:

Note, not only are the timestamps the same, and I did not do the two installs simultaniously, but the modifiersname is the replication process:
cn=replication dn,orclreplicaid=idmhost_oidrep,cn=replication configuration
Next step: install the Single Sign On and Delegated Administration Services

Update:

Starting up all processes (e.g. after a startup; I do not leave my test machines on 24*7), is as easy as 1-2-3:
Last login: Fri Jun 22 08:30:59 2007 from dbhost.home.local
[oidoracle@idmhost ~]$ opmnctl startall
opmnctl: starting opmn and all managed processes...
[oidoracle@idmhost ~]$ opmnctl status

Processes in Instance: rms.idmhost.home.local
-------------------+--------------------+---------+---------
ias-component      | process-type       |     pid | status
-------------------+--------------------+---------+---------
DSA                | DSA                |     N/A | Down
LogLoader          | logloaderd         |     N/A | Down
dcm-daemon         | dcm-daemon         |     N/A | Down
HTTP_Server        | HTTP_Server        |    3503 | Alive
OID                | OID                |    3518 | Alive

[oidoracle@idmhost ~]$ $ORACLE_HOME/ldap/bin/ldapcheck

Checking Oracle Internet Directory Processes ...ALL

Process oidmon is Alive as PID 3518
Process oidldapd is Alive as PID 3531
Process oidldapd is Alive as PID 3537
Process oidrepld is Alive as PID 3565
Not Running ---- Process odisrv

This odisrv is a bit of a nag. It is running perfectly on the other machine:
[oidoracle@oidhost ~]$ $ORACLE_HOME/ldap/bin/ldapcheck

Checking Oracle Internet Directory Processes ...ALL

Process oidmon is Alive as PID 3944
Process oidldapd is Alive as PID 3978
Process oidldapd is Alive as PID 3981
Process oidrepld is Alive as PID 4013
Process odisrv is Alive as PID 3983

However, opmnctl does not seem to control it, after a few stopall and startall, I had this:
[oidoracle@oidhost ~]$ $ORACLE_HOME/ldap/bin/ldapcheck

Checking Oracle Internet Directory Processes ...ALL

Process oidmon is Alive as PID 6410
Process oidldapd is Alive as PID 6411
Process oidldapd is Alive as PID 6426
Process oidrepld is Alive as PID 6585
Process odisrv is Alive as PID 6170
Process odisrv is Alive as PID 6414

Oh well. What bothers me is the fact odisrv does not run on idmhost; the log shows:
-----------------------------------------------------
Oracle Directory Integration Server instance# 01 started..
-----------------------------------------------------
Sat Jun 23 12:59:08 CEST 2007 : Starting Server to execute Profile Group :default against LDAP Server (idmhost.home.local:3130)
Sat Jun 23 12:59:09 CEST 2007 : SSL Mode :1
Sat Jun 23 12:59:09 CEST 2007 : Exception :javax.naming.AuthenticationException: [LDAP: error code 49 - Invalid Credentials]
Sat Jun 23 12:59:09 CEST 2007 : Aborting.. : null
Sat Jun 23 12:59:09 CEST 2007 : Exiting with Status -1: null

On odihost, the correct startup message in the log:
-----------------------------------------------------
Oracle Directory Integration Server instance# 01 started..
-----------------------------------------------------
Sat Jun 23 12:26:56 CEST 2007 : Starting Server to execute Profile Group :default against LDAP Server (oidhost.home.local:3130)
Sat Jun 23 12:26:56 CEST 2007 : SSL Mode :1
Guess I need to sort that out, before continuing to the next step.

Update: (don't try this - see below)
Change the port on idmhost.home.local from 389 to 3060, ran dcmctl updateconfig.
Then, I ran this, and all of a sudden, it worked!

[oidoracle@idmhost log]$ odisrvreg -D cn=orcladmin -w Welcome1 -p 3060
Registering for the first time...
DIS registration successful.

[oidoracle@idmhost log]$ $ORACLE_HOME/ldap/bin/ldapcheck

Checking Oracle Internet Directory Processes ...ALL

Process oidmon is Alive as PID 5645
Process oidldapd is Alive as PID 5648
Process oidldapd is Alive as PID 5660
Process oidrepld is Alive as PID 5697
Process odisrv is Alive as PID 5964

I'd have expected the odisrvreg utility to report "already registered - updating". This leaves a somewhat eery feeling; anyone knowing what is going on, please comment!
I'll update myself on that: the odisrv process does not need to run on both sides - it's supposed to failover. However, I still fail to see how - I even tried kill -9 (all processes), but could not get odisrv to start on the other node.

Let's continue with phase 7: installation of the middle tier:
Machines are fired up, all processes are up-and-running.
Phase 7a: Preliminaries (see phase 5a).
[root@idmhost ~]# groupadd idmown
[root@idmhost ~]# groupadd idminst
[root@idmhost ~]# useradd idmoracle -g idminst -G idmown -c 'Oracle Identity Mgmnt/SSO sw owner'
[root@idmhost ~]# passwd idmoracle
Changing password for user idmoracle.
New UNIX password:
Retype new UNIX password:
passwd: all authentication tokens updated successfully.
[root@idmhost ~]# mkdir /oracle/idm
[root@idmhost ~]# chown idmoracle:idmown /oracle/idm
[root@idmhost ~]# su - idmoracle
[idmoracle@idmhost ~]$ cp /install/Disk1/stage/Response/staticports.ini /oracle/idm/
edit staticports.ini: OID port: 3060, SSL OID port: 3130.

Phase 7b: Install first middle tier (SSO and DAS server).

Now, fire up Cywin X server, and:

frankbo@cs-frank03 ~
$ xhost +
access control disabled, clients can connect from any host

frankbo@cs-frank03 ~
$ ssh idmoracle@idmhost
idmoracle@idmhost's password:
Last login: Sun Jul 8 14:35:34 2007 from dbhost.home.local
[idmoracle@idmhost ~]$ export DISPLAY=192.168.1.104:0.0
[idmoracle@idmhost ~]$ /install/Disk1/runInstaller -invPtrLoc /oracle/idm/oraInventory/oraInst.loc

Fill in the correct settings:


Ditto:


It's still called "Infrastructure", although this is the middle tier:


And I still am not done with the Identity Management Install:


Oh, well, we've been here before...


So let's get started - note I added HA and Replication:


Select the correct file - it needs to pick up the ports actually in use by the OID install (phase 5)


This is an odd one: I am *not* adding a listener, so why this check is executed is beyond me. The resolution is to stop the services on this machine (logon as oidoracle, and issue an opmnctl stopall, or stopproc ias-component=OID)


Once the "error" hurdle is taken, select Cluster:


First install, so I have to create a cluster:


Name it:


Specify correct host; I had the "crossed" setup, so this SSO install (middle tier) will be served by the first install of the Infrastructure, which was on the oidhost:


Specify the password of orcladmin on the OID host:


I make a mistake here - specified the port, as used in metalink note 370458.1. Consequently, I had to change the loadbalancer:

balance -b login.home.local http idm1:7779 % idm2:7779 %


Make up a password, or -better yet- have one generated:


And finally - after a while, and the execution of the (in-)famous root.sh script:


This is what the last screen has to tell:
The following J2EE Applications have been deployed and are accessible at the URLs listed below.

Use the following URL to access the Oracle Enterprise Manager 10g Application Server Control Console :
http://idmhost.home.local:1156

The following information is available in:
/oracle/idm/idm10.1.2/install/setupinfo.txt

Oracle Application Server 10g (10.1.2.0.2) Usernames and Default password information:
Please refer to Oracle Application Server 10g Administrator Guide for more information.

Install Type: Identity Management
Configured Components: Oracle HTTP Server | Oracle Application Server Containers for J2EE | Oracle Application Server Single Sign-On | Oracle Application Server Delegated Administration Service | High Availability and Replication |

A new Oracle Application Server Cluster (Identity Management) has been created named SSOClusterA. The current instance has been joined this cluster at the end of installation.

Load Balancer Servers and ports specified for this instance:
HTTP Load Balancer: login.home.local:
LDAP Load Balancer: oidhost.home.local
SSL Port:3130
Non-SSL Port: 3060

Access URL for Oracle Delegated Administration Services for this instance:
http://login.home.local:80/oiddas

Administrator URL for Oracle Application Server Single-Sign On for this instance:
http://login.home.local:80/pls/orasso

Use the following URL to access the Oracle HTTP Server and the Welcome Page:
http://login.home.local:80
-----------------------------------------
Use the following URL to access the Oracle Enterprise Manager Application Server Control:
http://idmhost.home.local:1156

Instance Name: idm1012_01.idmhost.home.local

Installation of Oracle Application Server Infrastructure is Complete. Please note that any URLs created in this install may not be functional immediately.

Now - let me see if the loadbalancer works.

The defaul (login.home.local) Delegated Administration Service page:


After a successfull login:


After Logout, the node information is shown:


Ok - next step: phase 7c: passwordsI need to synchronize all passwords. One of the installation Wizards did randomize all passwords used in this setup. As connections may float, I do want passwords to be the same on both nodes. The script ssoReplSetup.jar is a Java script, residing in $ORACLE_HOME/sso/lib.
Update:

[oidoracle@oidhost ~]$ cd $ORACLE_HOME/sso/lib
[oidoracle@oidhost lib]$ export LD_LIBRARY_PATH=$ORACLE_HOME/lib32:$LD_LIBRARY_PATH
[oidoracle@oidhost lib]$ echo $LD_LIBRARY_PATH
/oracle/ias/oid10.1.2/lib32:/oracle/ias/oid10.1.2/lib
[oidoracle@oidhost lib]$ $ORACLE_HOME/jdk/bin/java -jar ssoReplSetup.jar -prompt
OracleAS Single Sign-On Replication Setup Tool
Release 10.1.2.0.0
Copyright (c) 2003, 2004 Oracle. All rights reserved.

Reading input paramterers ...

Enter MDS OID hostname : oidhost.home.local

Enter MDS OID port : 3060

Enter MDS OID administrator : cn=orcladmin

Enter MDS OID password : Welcome1

Enter MDS OID SSL Enabled (Y/N) : n

Enter RMS OID hostname : idmhost.home.local

Enter RMS OID port : 3060

Enter RMS OID administrator : cn=orcladmin

Enter RMS OID password : Welcome1

Enter RMS OID SSL Enabled (Y/N) : n

Enter RMS SYS DB password : MANAGER


Done reading parameters.

Contacting OID: ldap://oidhost.home.local:3060 ...
OID context received for MDS admin user, cn=orcladmin

Contacting RMS OID: ldap://idmhost.home.local:3060 ...
OID context received for RMS admin user, cn=orcladmin

MDS DB dn: orclReferenceName=DB1020.CS.NL,cn=IAS Infrastructure Databases,cn=IAS,cn=Products,cn=OracleContext
RMS DB dn: orclReferenceName=OIDREP.CS.NL,cn=IAS Infrastructure Databases,cn=IAS,cn=Products,cn=OracleContext
Starting password synchronization between MDS DB and RMS DB.
ERROR: RMS DB connection failed.
Action: Please check the RMS DB SYS Password.
Exception: java.sql.SQLException: ORA-28009: connection as SYS should be as SYSDBA or SYSOPER

java.sql.SQLException: ORA-28009: connection as SYS should be as SYSDBA or SYSOPER

at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:137)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:304)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:271)
at oracle.jdbc.driver.T4CTTIoauthenticate.receiveOauth(T4CTTIoauthenticate.java:647)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:307)
at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:433)
at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:150)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:31)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:512)
at java.sql.DriverManager.getConnection(DriverManager.java:171)
at oracle.security.sso.server.conf.SyncSSOPwd.syncUpPwds(SyncSSOPwd.java:303)
at oracle.security.sso.server.conf.SyncSSOPwd.main(SyncSSOPwd.java:752)

Checking the password revealed:
SQL> connect sys/manager@db1020 as sysdba
Connected.
SQL> connect sys/manager@oidrep as sysdba
ERROR:
ORA-01017: invalid username/password; logon denied
After changing it, I could logon as sysdba - the error is somewhat unclear - the message is right on spot:
[oidoracle@oidhost lib]$ $ORACLE_HOME/jdk/bin/java -jar ssoReplSetup.jar -prompt
OracleAS Single Sign-On Replication Setup Tool
Release 10.1.2.0.0
Copyright (c) 2003, 2004 Oracle. All rights reserved.

Reading input paramterers ...

Enter MDS OID hostname : oidhost.home.local

Enter MDS OID port : 3060

Enter MDS OID administrator : cn=orcladmin

Enter MDS OID password : Welcome1

Enter MDS OID SSL Enabled (Y/N) : n

Enter RMS OID hostname : idmhost.home.local

Enter RMS OID port : 3060

Enter RMS OID administrator : cn=orcladmin

Enter RMS OID password : Welcome1

Enter RMS OID SSL Enabled (Y/N) : n

Enter RMS SYS DB password : manager


Done reading parameters.

Contacting OID: ldap://oidhost.home.local:3060 ...
OID context received for MDS admin user, cn=orcladmin

Contacting RMS OID: ldap://idmhost.home.local:3060 ...
OID context received for RMS admin user, cn=orcladmin

MDS DB dn: orclReferenceName=DB1020.CS.NL,cn=IAS Infrastructure Databases,cn=IAS,cn=Products,cn=OracleContext
RMS DB dn: orclReferenceName=OIDREP.CS.NL,cn=IAS Infrastructure Databases,cn=IAS,cn=Products,cn=OracleContext
Starting password synchronization between MDS DB and RMS DB.
Creating RMS DB connection ... Done.

Synchronizing the password for orasso ...
MDS - orasso password: *****
Modifying orasso schema pwd value in RMS OID...
Modification of orasso user password in RMS OID successful.
Modifying the orasso user password in secondary database ...
Modification of orasso password in RMS db successful.

Synchronizing the password for orasso_ds ...
MDS - orasso_ds password: *****
Modifying orasso_ds schema pwd value in RMS OID...
Modification of orasso_ds user password in RMS OID successful.
Modifying the orasso_ds user password in secondary database ...
Modification of orasso_ds password in RMS db successful.

Synchronizing the password for orasso_pa ...
MDS - orasso_pa password: *****
Modifying orasso_pa schema pwd value in RMS OID...
Modification of orasso_pa user password in RMS OID successful.
Modifying the orasso_pa user password in secondary database ...
Modification of orasso_pa password in RMS db successful.

Synchronizing the password for orasso_public ...
MDS - orasso_public password: *****
Modifying orasso_public schema pwd value in RMS OID...
Modification of orasso_public user password in RMS OID successful.
Modifying the orasso_public user password in secondary database ...
Modification of orasso_public password in RMS db successful.

Synchronizing the password for orasso_ps ...
MDS - orasso_ps password: *****
Modifying orasso_ps schema pwd value in RMS OID...
Modification of orasso_ps user password in RMS OID successful.
Modifying the orasso_ps user password in secondary database ...
Modification of orasso_ps password in RMS db successful.

Setting SSO server preferences table in RMS DB ...
Connected to MDS DB as ORASSO user.
Retrieved SSO_SERVER pwd: *****
Decrypted SSO_SERVER pwd: *****
Connected to RMS DB as ORASSO user.

Setting SSO server preferences table in RMS DB ...
Connected to MDS DB as ORASSO user.
MDS node LDAP connection SSL usage: Y
ERROR: MDS node is configured to use LDAP over SSL.
ACTION: Please provide LDAP SSL port for the RMS node.

The last line indicates I should use the SSL port (3130):

[oidoracle@oidhost lib]$ $ORACLE_HOME/jdk/bin/java -jar ssoReplSetup.jar -prompt
OracleAS Single Sign-On Replication Setup Tool
Release 10.1.2.0.0
Copyright (c) 2003, 2004 Oracle. All rights reserved.

Reading input paramterers ...

Enter MDS OID hostname : oidhost.home.local

Enter MDS OID port : 3130

Enter MDS OID administrator : cn=orcladmin

Enter MDS OID password : Welcome1

Enter MDS OID SSL Enabled (Y/N) : Y

Enter RMS OID hostname : idmhost.home.local

Enter RMS OID port : 3130

Enter RMS OID administrator : cn=orcladmin

Enter RMS OID password : Welcome1

Enter RMS OID SSL Enabled (Y/N) : Y

Enter RMS SYS DB password : manager


Done reading parameters.

Contacting OID: ldap://oidhost.home.local:3130 ...
OID context received for MDS admin user, cn=orcladmin

Contacting RMS OID: ldap://idmhost.home.local:3130 ...
OID context received for RMS admin user, cn=orcladmin

MDS DB dn: orclReferenceName=DB1020.CS.NL,cn=IAS Infrastructure Databases,cn=IAS,cn=Products,cn=OracleContext
RMS DB dn: orclReferenceName=OIDREP.CS.NL,cn=IAS Infrastructure Databases,cn=IAS,cn=Products,cn=OracleContext
Starting password synchronization between MDS DB and RMS DB.
Creating RMS DB connection ... Done.

Synchronizing the password for orasso ...
MDS - orasso password: *****
Modifying orasso schema pwd value in RMS OID...
Modification of orasso user password in RMS OID successful.
Modifying the orasso user password in secondary database ...
Modification of orasso password in RMS db successful.

Synchronizing the password for orasso_ds ...
MDS - orasso_ds password: *****
Modifying orasso_ds schema pwd value in RMS OID...
Modification of orasso_ds user password in RMS OID successful.
Modifying the orasso_ds user password in secondary database ...
Modification of orasso_ds password in RMS db successful.

Synchronizing the password for orasso_pa ...
MDS - orasso_pa password: *****
Modifying orasso_pa schema pwd value in RMS OID...
Modification of orasso_pa user password in RMS OID successful.
Modifying the orasso_pa user password in secondary database ...
Modification of orasso_pa password in RMS db successful.

Synchronizing the password for orasso_public ...
MDS - orasso_public password: *****
Modifying orasso_public schema pwd value in RMS OID...
Modification of orasso_public user password in RMS OID successful.
Modifying the orasso_public user password in secondary database ...
Modification of orasso_public password in RMS db successful.

Synchronizing the password for orasso_ps ...
MDS - orasso_ps password: *****
Modifying orasso_ps schema pwd value in RMS OID...
Modification of orasso_ps user password in RMS OID successful.
Modifying the orasso_ps user password in secondary database ...
Modification of orasso_ps password in RMS db successful.

Setting SSO server preferences table in RMS DB ...
Connected to MDS DB as ORASSO user.
Retrieved SSO_SERVER pwd: *****
Decrypted SSO_SERVER pwd: *****
Connected to RMS DB as ORASSO user.

Setting SSO server preferences table in RMS DB ...
Connected to MDS DB as ORASSO user.
MDS node LDAP connection SSL usage: Y
Setting OID configurations in RMS DB Done.

Setting up the SSO Server site token in the prefs table...
Updating SSO preference store for the SSO Server site token...
SSO Replication configuration successfully finished.


Same thing needs to be done on the replicated site, idmhost.home.local. I found it not very clear whether this should be done in the middle tier, or in the infrastructure - the notes suggest the first, and so does the point in time: after the first middle-tier install.
Rest assured: it should run from the infrastructure - the sites, where the OID processes and replication run.
All that is left to install now, is the last middle tier:
[root@oidhost ~]# mkdir -p /oracle/idm/oraInventory
[root@oidhost ~]# cd /oracle
[root@oidhost oracle]# chown -R idmoracle:idminst idm


The following J2EE Applications have been deployed and are accessible at the URLs listed below.

Use the following URL to access the Oracle Enterprise Manager 10g Application Server Control Console :
http://oidhost.home.local:1156

The following information is available in:
/oracle/idm/idm10.1.2/install/setupinfo.txt

Oracle Application Server 10g (10.1.2.0.2) Usernames and Default password information:

Please refer to Oracle Application Server 10g Administrator Guide for more information.

Install Type: Identity Management

Configured Components: Oracle HTTP Server | Oracle Application Server Containers for J2EE | Oracle Application Server Single Sign-On | Oracle Application Server Delegated Administration Service | High Availability and Replication |

A new Oracle Application Server Cluster (Identity Management) has been created named SSOClusterB. The current instance has been joined this cluster at the end of installation.

Load Balancer Servers and ports specified for this instance:
HTTP Load Balancer: login.home.local:
LDAP Load Balancer: idmhost.home.local
SSL Port:3130
Non-SSL Port: 3060

Access URL for Oracle Delegated Administration Services for this instance:
http://login.home.local:80/oiddas

Administrator URL for Oracle Application Server Single-Sign On for this instance:
http://login.home.local:80/pls/orasso

Use the following URL to access the Oracle HTTP Server and the Welcome Page:
http://login.home.local:80
-----------------------------------------
Use the following URL to access the Oracle Enterprise Manager Application Server Control:
http://oidhost.home.local:1156

Instance Name: idm1012_02.oidhost.home.local

Installation of Oracle Application Server Infrastructure is Complete. Please note that any URLs created in this install may not be functional immediately.


The installation is the same as the first one, except for some names, that are different (obviously): the cluster is called SSOClusterB (could have been the same, by the way), the ldapserver is idmhost.home.local (I am installing on oidhost!), so I will not post any screendumps of that.
Instead, stay tuned for replication woes, and usage notes.

Last and Final Update:
To show that the whole things is two-fold:


There you have it - two partner applications.

In a nutshell:
  1. Install and patch the database software tree(s).
  2. Create a database, altering default the settings to ones, fit for a Repository. If not done now, the Metadat Creation Repository Assistant (MRCA) will force you.
  3. Run the MRCA against the newly created database.
  4. Clone to create the replication database (or reuse the scripts and rerun MRCA)
  5. Install the first Infrastructure. Options: OID and DIP. Use main database as repository database.
  6. Install the second Infrastructure. Options: OID, DIP and HA/Replication, use first infrastructure OID setup as reference. Use replica database for repository database.
  7. Configure your network:
    1. make sure you can start SQL*Plus from both database, and both Infrastructure environments. Als make sure, you can use shorthand, as well as the fully qualified tns-alias. This step is crucial!
    2. Also, make sure you have your loadbalacer and naming (DNS or other) in order.
  8. Setup the OID replication, using the remtool ($ORACLE_HOME/bin/remtool -asrsetup -v)
  9. Stop and start (using $ORACLE_HOME/opmn/bin/opmnctl) all processes on both Infrastructure installations.
  10. Start the replication processes; first time only by hand, using oidctl, on both Infrastructure installations.
  11. Check replication by adding on an entry in one OID environment, wait until it appears in the other. then, delete from the other, and check whther it disappears from the first.
  12. Install the first Middle Tier (Single Sign On/Delegated Administration Services). Oddly enough, it is still an infrastructure install. Select SSO, DAS and HA, create a new cluster. Specify the first OID install for LDAP, and your loadbalancer.
  13. Synchronize passwords, generated at random during the installation, across both infrastructures. Use ssoReplSetup.jar -prompt on both Infrastructure installs. Mind the LD_LIBRARY_PATH.
  14. Install the second Middle Tier (SSO/DAS). Similar to first install.
Can be done in two days, provided everything is prepared, root access available and no platform specific bugs are ran into. And enough coffee...

Thursday, June 07, 2007

A test bed, or how to install Linux fast!

How do you prepare for Oracle testing? Or anything else, for that matter, based on Linux? Here's how:

Preparation:
  1. Hardware. Of course, you should have the hardware. I have.
  2. Download the CentOS 4.4 Server CD; it can be found here.
  3. Burn the .iso on a CD - any burning program can do that, many recognize the downloaded file as an image.
The next steps take about twenty minutes, even less on a modern CPU.
  1. Boot the designated system from newly created CD, some systems may require fiddling the boot options in the BIOS to allow for CD-drives to be searched for bootable products. Consult your motherboard manual if your system will not boot from the CD.
    Once it booted, you will see a startup screen, type:
  2. linux text
  3. Skip the CD test
  4. Choose English for a language, and a US Keyboard (or your own favourite, of course!)
  5. Choose to Auto Partition the disks, and remove any partitions (if applicable).
  6. OK the default partitioning offered (this is not about optimal Linux, but Oracle installations)
  7. Use Grub as bootloader, no password, no special config options
  8. Allow the Master Boot Record to boot from.
  9. Use a fixed IP-address: I used 192.168.1.210. Servers should always have fixed IP-addresses.
  10. As for Default gateway and DNS servers: use your own network settings. You could type:
    ipconfig/all
    on a windows PC, connected to the internet to find out your settings (assuming you have both a server, which now is cleaned out, and a workstation).
  11. Hostname: anything.home.local (no, choose your own, but do not use a domain, that might exist now or in the future - the domain home.local will never exist)
  12. Choose to use No firewall (yes, sure about that). Not that you cannot get Oracle installations to work with a firewall, but not having to bother with ports and firewall configurations while installing Oracle stuff makes the install process so much easier. Besides - you do have a central firewall in your network, don't you? If you don't: disconnect the RJ45 Ethernet connector NOW!
  13. SE Linux level: select to warn
  14. No additional languages (or: what you want)
  15. Indicate correct timezone
  16. Package install: select Customize.Leave the defaults for the following parts as is, uless otherwise mentioned. Deselect any other!
  • X-Windows
  • Gnome
  • Server Config Tools: add Samba (unless you want FTP - then add that)
  • Windows File Server (unless you like FTP)
  • Admin tools
Wait for 8 minutes or so, then the server will reboot. You have now completed your CentOS 4.4 install - congratulations!

Next step - upgrade.
The time, next step will take heavily depends on the type of network connection your have; it took me about half an hour to complete, with a 4Mbps downstream connection.
See if you can ssh into your server - I like TeraTerm. There seem to be new versions out, just search the net. You may have to use the ip-address you used (I used 192.168.1.210, remember?) until changes are made to your local hosts file.
Log on as root, and issue:

[root@oidhost ~]# uname -a
Linux oidhost.home.local 2.6.9-42.EL #1 Sat Aug 12 09:17:58 CDT 2006 i686 athlon i386 GNU/Linux
[root@oidhost ~]# yum upgrade

This will take some time, but when asked, answer Y - it will replace 155 of the installed packages (and upgrade to CentOS 4.5 - that distro does not have a Server Only iso, though...)
When you get the warning:

rpmts_HdrFromFdno: V3 DSA signature: NOKEY, key ID 443e1821
Public key for net-tools-1.60-37.EL4.9.i386.rpm is not installed
Retrieving GPG key from http://mirror.centos.org/centos/RPM-GPG-KEY-centos4
Importing GPG key 0x443E1821 "CentOS-4 key "
it is OK: type 'Y' and continue.
When yum finally has completed:

[root@oidhost ~]# reboot -n

After the boot, we'll have a new kernel:

[root@oidhost ~]# uname -a
Linux oidhost.home.local 2.6.9-55.EL #1 Wed May 2 13:52:16 EDT 2007 i686 athlon i386 GNU/Linux


That's it! You now run CentOS 4.5 - congratulations!

Additions:
1. maintenance

There's a neat, web based administrative package, I'm rather fond of: webmin. Installing is easy as 1-2-3:
Visit the webmin site, and find the url of the download mirror nearest to you - I actually needed to start the download, in order to get the url copied. Using the url, that is under "Download" will do redirects, and the rpm installer does not understand them.
I use the Amsterdam mirror (By the way: that is one line you need to feed rpm with):

[root@oidhost ~]# rpm -iv http://surfnet.dl.sourceforge.net/sourceforge/webadmin/webmin-1.340-1.noarch.rpm
Retrieving http://surfnet.dl.sourceforge.net/sourceforge/webadmin/webmin-1.340-1.noarch.rpm
warning: /var/tmp/rpm-xfer.cxUGCI: V3 DSA signature: NOKEY, key ID 11f63c51
Preparing packages for installation...
Operating system is CentOS Linux
webmin-1.340-1
Webmin install complete. You can now login to http://oidhost.home.local:10000/
as root with your root password.

Yeah - right, not until I changed my Windows hosts file... which, by the way, has the following contents:
127.0.0.1       localhost
192.168.1.210 oidhost.home.local oidhost
192.168.1.220 idmhost.home.local idmhost
192.168.1.104 infradbhost.home.local infradbhost
The first entry is default, and best left alone (your loopback connector needs this!)
The second and third are my application server layers to be, and the last entry is my workstation, where the databases reside. Runs XP, and has 3GB memory available.

OK, after the alteration you can actually log in to webmin (and TeraTerm) using the hostname, not the IP-address.
Locate the Bootup and Shutdown section (under System), and stop and disable:
sendmail, cups, kudzu, pcmcia, xinetd.
Of course, you could have entered:
chkconfig sendmail off
chkconfig cups off
chkconfig kudzu off
chkconfig pcmcia off
chkconfig xinetd off
from your SSH session... followed by
service sendmail stop
service cups stop
service xinetd stop

2. Allowing copies from your workstation
Now, I installed Samba, you may have chosen FTP Server. If so, configure it. I'l do the Samba Windows thing, from WebMin:
  1. select Servers, Samba Windows file sharing.
  2. open the Windows Networking options, and alter:
  3. Workgroup - to whatever you use, the Windows default is WORKGROUP. You may leave it at WORKGROUP.
  4. You may alter the Server Description and Server name, leaving them default will cause misunderstanding with the second server in use...
    I used the hostname, oidhost and for the second machine: idmhost.
  5. Chance security to Share
  6. SAVE.
Under the shares, Create a New one, and call it install. Directory: /install.
Now click on the 'Start Samba Server' button, and browse your network - you should be able to see the install share from your windows client.
You will be asked for a password (and a user), whcih we still need to setup:
[root@oidhost ~]# useradd frank
Now - in WebMin, click on the 'Convert Unix users to Samba users', leave everything default,
and verify that frank is added after clicking the 'Convert' button. Then setup the
password that will be used for the share:
[root@oidhost ~]# smbpasswd frank
New SMB password:
Retype new SMB password:

You should now be able to map \\oidhost\install to a drive, using frank as user (select 'Connect as a different user'), and using the supplied password.
You still can't do a thing, as security prohibits.
For the sake of speed (again: we're in it for Oracle, not Linux):
chown frank /install
You now own the bloody thing!
Copy over any .cpio files and patches.

You are now ready to install Oracle - congratulations!
Addition: Not quite... some extra packages are required, better install these now:
[root@oidhost ~]# yum install xorg-x11-deprecated-libs.i386
[root@oidhost ~]# yum install gcc gcc-c++ openmotif21 sysstat gnome-libs libstdc++-devel compat-libstdc++-296 compat-db-4
After installing these, I noticed I had to set the kernel params; here's how:
edit /etc/security/limits.conf, and add this line:
*       hard    nofile  65536
Edit /etc/sysctl.conf, and add:
# Oracle specifics:
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
# semaphores: semmsl, semmns, semopm, semmni
kernel.sem = 256 32000 100 142
fs.file-max = 131072
net.ipv4.ip_local_port_range = 10000 65000
kernel.msgmni = 2878
kernel.msgmax = 8192
kernel.msgmnb = 65535

A reboot is required...

Wednesday, April 04, 2007

Oracle Apex goes commercial!

Yesterday, Oracle launched version 3 of Application Express, also known as ApEx. I wasn't sure about the new frills it would contain - PDF from within wasn't something I was missing. After all, it has been possible to produce PDF since version 1.5.

But things sure have been improved - PDF printing is now so much easier!
Edit: I fell for it. I'll admit; I fell for the marketing machine.
PDF printing actually is the same as it ever was: it uses the Open Source Java printing engine, Apache FOP. The 'easy' way would be to buy the BI suite from Oracle. So I changed the title into Apex goes commercial. Quite disappointed, or rather, mad with myself, so not reading better.

Oracle 11g.

And ApEx is Oracle 11 compliant - wait till that hits the streets. So much is new for Oracle 11 (still the Grid verion: 11g): it has an embedded pl/sql gateway, which will be used as default.

Installations with an application server and using mod_plsql will still be possible; they just will not be the default. Looks like ApEx is becoming more of a database centric tool - it does not need the iAS anymore.

Not that it actually did - ever noticed how ports 8080 and 2100 are opened for http and ftp protocols respectively after a default install? XDB is accountable for that.

Features

If you have never taken a look at ApEx, do so now. My favourites are the Webservices (now JDeveloper and BPEL are supported), password and account control enhancements (expiry, grace time, forced changes upon first login). Some people will probably like the flash charts.

I will install 3.0 in a develop environment tomorrow, and see if I can find the time to mess around with it.
Edit: Of course not... The day started off with a webservice, not performing (due to a wrongly configured log4j), continued with jobs that needed killing or scheduling, software that had to be tested and rolled out, production crash actions involving InterConnect, and before you know it, you're in the car back home, trying to aviod the traffic jam, caused by accidents, and closed highways...

Wednesday, December 27, 2006

Who said I was?

I already noted the nasty side effects, but instead of altering the once published blog, I decided to add this one. After all, the portal single-sign on (SSO) was just an example, and I want the originally registered applications still to be registered, including OIDDAS, orasso and both the infrastructure as well as the middle tier Oracle HTTP Servers (or OHS).

Inventory taking.
What is it I will have to move? Let's check with the Single Sign-On:orasso entry on the Infrastructure management console. Opening sub pages will get me to the Orasso SSO home page, in this exercise located at http://asinfra101202.mymachine.local:7777/pls/orasso/orasso.home.
This page provides a list of all current "partner applications" that rely on orasso to take care of the login procedure.
After logging in (with orcladmin) by clicking the modest link in the right upper hand corner, you will be shown another link: "Maintain the SSO server". Particulary interesting is the link about maintenance on partner applications!

History
Was did I already do? Actually, not a lot - the changing of the DAD is what was done, to allow the midtier to access /pls/orasso, but that was about it. Reread that thread first.
In the mean time, I do have access to the orasso home page, using the midtier link as well; however, the login link provides me with an error, and all page links refer to the infrastructure.

Moving the SSO server
Less of a move than fact - only the references will be changed to the middle tier instance, not the actual server.
First of all, I want to get rid of the references to the infrastucture on the orasso home page. I can do that by changing the home URL, success URL and logout URL in the Maintain Partner Applications screen of orasso. Unfortunately, I cannot change the login URL, nor the Single Sign-Off URL on this page, and those still point to the infrastructure.
So, the second step is to reregister orasso, by running this on the infrastructure: $ORACLE_HOME/sso/bin/ssoreg.sh -oracle_home_path $ORACLE_HOME -site_name "The SSO Server (OraSSO)" -config_mod_osso TRUE -mod_osso_url http://asmt101202.mymachine.local:7777/pls/orasso/orasso.home


After that, stop and start all opmn registered processes - both sides!

Friday, December 15, 2006

So, you're the security expert on Oracle iAS... or How To Relocate the SSO server

Well, uhm, actually I'm not, but how often do you find yourself in the position that the consulting firm you work for positions you as one?
Picture the following: a perfectly reasonable setup of Oracle's internet Application Server, with Oracle's LDAP implementation (Oracle Internet Directory, or OID) and Portal, Single Sign On (SSO), the Oracle Certificate Authority (OCA), Forms, Discoverer, the Oracle HTTP Server (OHS) and WebCache.
Basically, it looks like this:


What runs where?
Here's what the documentation has to say about the middle or Application Server tier:
The middle-tier is the part of an Oracle Application Server architecture that contains several components responsible for accepting requests from clients, validating the requests, and providing content, while using intelligent data caching for faster and reliable performance. For OracleAS Portal, the middle-tier handles all Web requests by forwarding them to the appropriate provider. This is also where Portal pages are assembled, and where the caching of Portal content is managed. The middle-tier also provides other functions for other Oracle Application Server components.

Components on the Middle tier are:
  • The Oracle HTTP Server (powered by Apache), or OHS
    OHS handles all requests for Portal, either via mod_plsql, or via the Parallel Page Engine (PPE)
  • Webcache
  • the Application Server containers for J2EE (OC4J)
Same for the Infrastructure tier:
By default, the infrastructure tier handles all authentication requests and hosts the Oracle Application Server Metadata Repository, which contains schemas and business logic used by application server components (including OracleAS Portal) and other pieces of the infrastructure.
Mind the "By default"...

Components, running in the infrastructure are:
  • Application Server Control
  • Oracle Internet Directory (OID)
  • Oracle Application Server Single Sign on (SSO)
  • Oracle Application Server Metadata Repository
    some middle tier products, like Portal, store their metadata in this repository.
Simple enough, isn't it? Sure, until the network guys came in, as they demanded -and rightly so- this:

So - what's the problem? The middle tier is moved to the DMZ, and firewalls are in place. What can be the problem? Well, Portal doesn't work anymore, as nobody can login...
How comes?

A closer look.
First of all, Portal "runs" on the middle tier; the OHS on the middle tier uses mod_plsql and/or the Parallel Page Engine. Do'nt be mislead by Portal pages themselves, as there is a rewrite engine. A portal page reference (before the rewrite!) may look like:
http://asmt101202.mymachine.local:7777/pls/portal
So, there must be a DAD defined on the middle tier, because of the /pls/ :
[oracle@asmt101202 bin]$ cat $ORACLE_HOME/Apache/modplsql/conf/dads.conf
<Location /pls/portal>
SetHandler pls_handler
Order allow,deny
Allow from All
AllowOverride None
PlsqlDatabaseUsername portal
PlsqlDatabasePassword @BW01fa60ocxhrL9E2N88jEyS0bXgwimw==
PlsqlDatabaseConnectString cn=asdb,cn=oraclecontext NetServiceNameFormat
PlsqlNLSLanguage AMERICAN_AMERICA.AL32UTF8
PlsqlAuthenticationMode SingleSignOn
PlsqlDocumentTablename portal.wwdoc_document
PlsqlDocumentPath docs
PlsqlDocumentProcedure portal.wwdoc_process.process_download
PlsqlDefaultPage portal.home
PlsqlPathAlias url
PlsqlPathAliasProcedure portal.wwpth_api_alias.process_download
</Location>

So, there you have it - on the middle tier, there's a Database Access Descriptor, pointing to the (standard infrastrutcure setup named) database "asdb". Whenever someone logs on to Portal, SSO is activated, the login link on Portal looks like
http://asmt101202.mymachine.local:7777/pls/portal/PORTAL.wwsec_app_priv.login?p_requested_url=http%3A%2F%2Fasmt101202.mymachine.lo....

Following that link, you will get redirected to
http://asinfra101202.mymachine.local:7777/sso/jsp/login.jsp?site2pstoretoken=v1.2~458......

and that is where the trouble starts: asinfra101202 is not in the DMZ, it's host name cannot be resolved by the DNS-server, in short: you'll get a "page not found" response.

What SSO does.
The Single Sign On server uses mod_osso, which is configured on the infrastructure server. The flow of events is as follows:
  1. The user requests a page; the Apache configuration is such that authentication is needed.
  2. The mod_osso module kicks in, and looks for a cookie with an encrypted token. If found, the token is passed back to the application.
    If not found, the request gets redirected to the SSO server.
  3. The SSO server requests it's own cookie - if not found, it produces a login page.
  4. The user credentials are checked and validated against OID
  5. The SSO server sets it's own cookie, and redirects the user identity in encrypted form back to mod_osso.
  6. mod_osso sets it's own cookie, with a token, which holds the user identity in encrypted form (see step 2)


Also, one needs to realize that Oracle Portal is considered a "partner application".
SSO used to be incorporated in Portal, hence the -incorrect- saying 'portal is a partner app of itself'.
Not since iAS 9.0.2, where SSO and Portal were disconnected (and mod_osso was born).




What to do?
Now that the problem is clear, so is the solution. Basically, we need to reconfigure mod_osso, so that:
  1. it can be accessed from the middle tier
  2. redirection takes place to the middle tier
The SSO server Home Page is located at [host][:port]/pls/orasso, where the host is your infrastructure server, e.g.
http://asinfra101202.mymachine.local:7777/pls/orasso/orasso.home
We will have to change that for starters, so that is will become
http://asmt101202.mymachine.local:7777/pls/orasso/orasso.home

As this is a /pls/ entry, too, it must be a DAD, just like Portal uses, so let's check the dads.conf file on the infrastructure machine:
[oracle@asinfra101202 ~]$ cat $ORACLE_HOME/Apache/modplsql/conf/dads.conf
#=========================================================== # mod_plsql DAD Configuration File
# ===========================================================
# 1. Please refer to dads.README for a description of this file
# ===========================================================
# Note: This file should typically be included in your plsql.conf file with
# the "include" directive.
# Hint: You can look at some sample DADs in the dads.README file
# ===========================================================
<Location /pls/orasso>
SetHandler pls_handler
Order deny,allow
Allow from All
AllowOverride None
PlsqlDatabaseUsername orasso
PlsqlDatabasePassword @BcyVNZv4W9rd5+bPajizTuISOz9AvJ3lLg==
PlsqlDatabaseConnectString cn=asdb,cn=oraclecontext NetServiceNameFormat
PlsqlNLSLanguage AMERICAN_AMERICA.AL32UTF8
PlsqlAuthenticationMode SingleSignOn
PlsqlSessionCookieName orasso
PlsqlDocumentTablename orasso.wwdoc_document
PlsqlDocumentPath docs
PlsqlDocumentProcedure orasso.wwdoc_process.process_download
PlsqlDefaultPage orasso.home
PlsqlPathAlias url
PlsqlPathAliasProcedure orasso.wwpth_api_alias.process_download
</Location>

Just add that to the existing DADs in the dads.conf file on the middle tier, and bounce the OHS:
[oracle@asmt101202 bin]$ opmnctl restartproc type=ohs
opmnctl: restarting opmn managed processes...


Then, try to access the SSO Server's home page:
http://asmt101202.mymachine.local:7777/pls/orasso/orasso.home

That works...[edit]But you cannot login; you will get: WWC-41439.[/edit]
However, the Portal/SSO login page still redirects you to the infrastructure. You will have to reregister the SSO server. Now, there are two ways of doing that:
  1. Call the ssoreg.jar in $ORACLE_HOME/sso/lib
  2. use the ssoreg script (or batch, when using MS), located in $ORACLE_HOME/sso/bin.
Some posts suggest to use the latter, as it sets some environment variables; so I will. What this registration does is create an encrypted osso.conf file.
[edit]There are some nasty side-effects... See Metalink Note:249408.1 What will happen, is that an empty obfuscated configuration file, osso.conf, is created, to which Portal is added. Portal may work OK after these actions, other vital SSO applications, like OIDDAS, will not.[/edit]
[oracle@asmt101202 bin]$ $ORACLE_HOME/sso/bin/ssoreg.sh -oracle_home_path $ORACLE_HOME -site_name asmt101202.mymachine.local:7777/pls/portal -config_mod_osso TRUE -mod_osso_url http://asmt101202.mymachine.local:7777/pls/portal

I did it on the infrastructure tier as well, seemed logical, but I have no real explanation as to why. I do remeber a note on metalink, mentioning it should be done on both.
Make sure all configs are updated: $ORACLE_HOME/dcm/bin/dcmctl updateconfig -d -v
stop and start all processes, both ends: opmnctl stopall, opmnctl startall
You will stil get an error page; make sure to alter the iasadmin.xml file on the middle tier ($ORACLE_HOME/portal.conf/iasadmin.xml - alter the Host entry on line 2), and run ptlconfig:

oracle@asmt101202 conf]$ ./ptlconfig -dad portal -site -pw welcome2
~/product/as101202/midtier/portal/conf ~/product/as101202/midtier/portal/conf

Portal Dependency Settings Tool

Processing Portal instance '/pls/portal' (cn=asdb,cn=oraclecontext)
Processing complete

Now - following the "login" link on the orasso home page, I get:
You cannot login because there is either invalid or no configuration information stored in the enabler configuration table (WWSEC_ENABLER_CONFIG_INFO$ (WWC-41439))
The WWC-41439 is a link to an error page, that tells me:
WWC-41439 - You cannot login because there is either invalid or no configuration information stored in the enabler configuration table (WWSEC_ENABLER_CONFIG_INFO$).
Cause:
One or both of the following occurred:
  • An alias defined in the Apache configuration caused Apache to translate host.domain.com to just host. If this is the case, the Login link only shows host:port (dropping the domain).
  • The default domain was not set in the Apache configuration. When this occurs, only the hostname is shown in the Login link and the domain is not included.
  • The Portal was configured with an incorrect host or port.
Action:
Do one or both of the following:
  • Remove all such aliases from your Apache configuration.
  • Include the domain in the ServerName parameter.
  • Fix the Host in the IASInstance element and ListenPort in the WebCacheComponent element in iasconfig.xml and run ptlconfig -dad portal -site. The ptlconfig script and the iasconfig.xml file is normally located in the directory portal/conf under the OracleAS Portal and OracleAS Wireless middle-tier home.

Looks like we still have to explain to portal, that the url's are routed somewhere else, which makes sense, of course.
Just fixing the host entry in iasconfig.xml and running the ptlconfig utility does not seem to be enough; inspecting the table shows:
SQL> select LS_LOGIN_URL from portal.WWSEC_ENABLER_CONFIG_INFO$
/

LS_LOGIN_URL
------------------------------------------------------------------------------------------------------------------------------------
http://asinfra101202.mymachine.local:7777/pls/orasso/orasso.wwsso_app_admin.ls_login

SQL> update portal.WWSEC_ENABLER_CONFIG_INFO$
2 set LS_LOGIN_URL = 'http://asmt101202.mymachine.local:7777/pls/orasso/orasso.wwsso_app_admin.ls_login';

1 row updated.

SQL> commit;


That does the trick - no more redirects.
In a nutshell:
  1. Enable the orasso DAD on the middle tier.
  2. Reregister the SSO server (middle tier)
  3. Fix iasconfig.xml
  4. Update the portal table

Thursday, December 14, 2006

Troubles....

About two weeks ago, my monitor went dead. I switched it on, and it did ''poof!" - lights out. Funny smell came from it, too. Anyway - no fuse to be seen, and as the main switch was dead as well (no power light), I decided to say farewell to my 19" Iiyama, that has served me for about 8 years.
So - something new had to be bought, but what? I wanted a 22" wide Samsung (the SyncMaster 225 BW), but after discussing the pro and cons, I decided for a 204B (20", 1600 by 1200 resolution), and a graphics card upgrade (my old card could neither handle the resolution, nor DVI - not that DVI is a requirement...).
Luckily, the "new" card fitted in my computer (which still uses AGP - I know, hopelessly old fashioned), so I'm typing away with a crisp, clear screen, and a lot more space on my desk.