Wednesday, June 27, 2007

Handy

these plugins, for Firefox users. Search Metalink, OTN, documentation, whatever.
Update: The Oracle toolbar has a new version. Still not published on OTN, but you can find it here.

Sunday, June 10, 2007

High performance, High availablity in Oracle Application Server

Last update: Aug, 10, 10:53 (11g Download!)Ambitious?
Here is what I want to do. I have done setups according to the Enterprise Deployment Guide, ending up with a configuration similar to what you can see here, and I've also witnessed stuff, described in numerous metalink articles (so it must be hard :) ) and in the High Availability Guide, that lead to this.
Now, for some reason, my current assignment does not allow RAC setups. Reasoning fails, but I have given up after a year-and-a-half: "This is the first release of RAC - we don't do first releases". Which is crap, of course , since the predecessors have been around some 8 (yes - eight) releases: 7.1, 7.2, 7,3, 8.0, 8.1, 9.0, 9.1, 10.1 and now 10.2. Thanks to Oracle Marketing...
Anyway, that leads to a point where the database has become the single point of failure, when using the Enterprise setup. This setup uses the same (Clustered) database as OID storage, as well as the Application Server Repository.
Using the High Availability Guide, you will not have High Performance: as one link in the chain breaks, the whole chain is unavailable - you will still have the parallel chain, so availability does not suffer, just performance. This is due to the fact the databases do act as backups for OID, but not for the Application Server Metadata!
So, where the first setup is clustered on Application Server level, the second is not. Where the second setup allows one chain to become completely, or partially unavailable, the first approach will fail in the database department (which is not, and cannot be RAC!). What I want is best of both! I want Application Server Clustering, and Load Balancing, and Replication and Fail-Over! So, there you have it.
Ambitious? Sure!
Can it be done? Well, I actually don't know. You are here to find out.
When do you know? Well, at the end of the story, and this may become a lengthy one. I do not want to split is, as I did with the Enterprise Security entries, so I will update this article as I go.

Preparations
I have to my disposal three machines, all equipped with two harddisks. The latest replacement is equipped with an Intel E6600 processor, 4GB ram and two 320GiB SATA disks. It replaced the AMD 2100+ with 1.5GB memory and two 80GB ATA100 disks in a stripe set.
For test purposes, I already had a "server", which has been used before.
Both machines have been rebuilt, using the previous post.
All three machines are interconnected via a gigabit switch, using proper, short, 1GB certified network cables. The gigabit ethernet interfaces are the onboard ones, and price of the switch is not an issue anymore.
So much for the hardware; as for the software, better start early, as there is some 6,832MB (well over 6GB!) to be downloaded! Of course, the 400MB from CentOS is already completed. That leaves:
  • Downloads from oracle:
  1. 10G release2 Database (640MB for the Windows version)
  2. 10G Release2 Companion (another 640MB for the Windows version)
  3. Patch 10.2.0.3 (almost 900MB for Windows)
  4. CPU Apr2007 for the database (another 140MB)
  5. iAS 10G release 2 (2 GB in 4 cpio files)
  6. iAS 10G release 2 patchset (4960210 - 1.7GB)
  7. CPU Apr2007 patches for Identity Management and OID installs - some 12 MB
  8. Metadata Repository Creation Assistant V10.1.2.0.2 (400MB zipfile)
    I chose the Windows version; basically it does not matter, it executes against remote databases
  • Balance (http://www.inlab.de/balance.html)
Yes, I am going to do the load balancing as well - just take a look at either the enterprise setup or the high availability setup: both have loadbalancers. Also, these beasts have names. Now, I may have a few buck to spare for a gigabit switch, buying loadbalancing routers is a tad too much. So I will use non-dedicated software based (as opposed to firmware) load balancing. Should be interesting!

Installation
Once done downloading the software, and redistributing over all systems (see the previous post), I started installing.

Installation phase 1: the databases

Well, nothing much to tell about installing that base and patch level on Windows, but for some tricks:
  1. Install the baseline version of the software, do not create a database, or select a prebaked one.
  2. Install ultraSearch from the Companion CD.
  3. Patch software, twice (patches 5337014 and 5948242)
Only then fire up the Database Creation Assistant. Make sure you have sga_target, java_pool_size, processes and some other stuff increased, or you wil have to do it later when creating the Repository! Here's my init.ora with correct settings:
db_block_size=8192
db_file_multiblock_read_count=16

open_cursors=300

db_domain="home.local"
db_name=db1020

background_dump_dest=D:\oracle\admin\db1020\bdump
core_dump_dest=D:\oracle\admin\db1020\cdump
user_dump_dest=D:\oracle\admin\db1020\udump

control_files=("D:\oracle\oradata\db1020\control01.ctl", "D:\oracle\oradata\db1020\control02.ctl", "D:\oracle\oradata\db1020\control03.ctl")

job_queue_processes=10
compatible=10.2.0.3.0

processes=400

sga_target=600M

audit_file_dest=D:\oracle\admin\db1020\adump
remote_login_passwordfile=EXCLUSIVE
pga_aggregate_target=122683392
db_cache_size=144M

undo_management=AUTO
undo_tablespace=UNDOTBS1

aq_tm_processes=2
shared_pool_size=175M
java_pool_size=120M

Installation phase 2: create the Repository
Unpack the zip file, and install the Metadata Repository Creation Assistant. After that, just run the bloody thing - not much to tell here, apart from the strange behavior where 23 datafiles, totalling 1.4GB of diskspace gets written, deleted and written again. I chose to have the repository related files all in one location (hey - this is just a demo!), but separate them from the other database datafiles, by using a one level deeper subdirectory "rep".
If the checks on paramaters fail, alter them. This is a fairly easy install.

Installation phase 3: prepare for Replication - create the second instance
I used RMAN clone database for this. I need a (clean) backup anyway, so here we go. There are two stages: making the backup, and restoring the clone:

Phase 3a: backup.

Open a Command Line Interface (MS Windows: Run-> cmd, *ix: your favorite shell)
set your environment variables
RMAN target /
shutdown
startup mount
backup database;

Phase 3b: clone.
Create the (empty) directories for the clone (data/admin)
Copy init.ora and alter directory paths and instancename
Add newly created instance to tnsnames.ora and listener.ora.
Start listener.
MS Windows only: create the service:
Open a Command Line Interface (MS Windows: Run-> cmd)
oradim -new -sid oidrep -pfile D:\oracle\admin\oidrep\pfile\initoidrep.ora

Create passwordfile:

orapwd.exe file=%ORACLE_HOME%\database\PWDoidrep.ora password=oracle force=y
set oracle_sid=oidrep
sqlplus / as sysdba
startup nomount pfile=D:\oracle\admin\oidrep\pfile\initoidrep.ora
exit

Let's clone!
set oracle_sid=db1020
rman
connect target /
connect auxiliary sys/oracle@oidrep.home.local

duplicate target database to oidrep
pfile=D:\oracle\admin\oidrep\pfile\initoidrep.ora
db_file_name_convert=(
'D:\oracle\oradata\db1020', 'D:\oracle\oradata\oidrep',
'D:\oracle\oradata\db1020\rep', 'D:\oracle\oradata\oidrep\rep')
logfile 'D:\oracle\oradata\oidrep\redo01.log' size 100M,
'D:\oracle\oradata\oidrep\redo02.log' size 100M,
'D:\oracle\oradata\oidrep\redo03.log' size 100M;

That is it! This is a screen scrape from the actual session (some lines are snipped for brevity):
C:\Documents and Settings\frankbo>oradim -new -sid oidrep -pfile D:\oracle\admin\oidrep\pfile\initoidrep.ora
Instance created.

C:\Documents and Settings\frankbo>orapwd.exe file=%ORACLE_HOME%\database\PWDoidrep.ora password=oracle force=y

C:\Documents and Settings\frankbo>set oracle_sid=oidrep

C:\Documents and Settings\frankbo>sqlplus / as sysdba

SQL*Plus: Release 10.2.0.3.0 - Production on Sun Jun 3 13:38:53 2007

Copyright (c) 1982, 2006, Oracle.  All Rights Reserved.

Connected to an idle instance.

SQL> startup nomount pfile=D:\oracle\admin\oidrep\pfile\initoidrep.ora
ORACLE instance started.

Total System Global Area  629145600 bytes
Fixed Size                  1292132 bytes
Variable Size             318769308 bytes
Database Buffers          301989888 bytes
Redo Buffers                7094272 bytes
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Production
With the Partitioning, OLAP and Data Mining options
C:\Documents and Settings\frankbo>set oracle_sid=db1020
C:\Documents and Settings\frankbo>rman
Recovery Manager: Release 10.2.0.3.0 - Production on Sun Jun 3 13:41:49 2007
Copyright (c) 1982, 2005, Oracle.  All rights reserved.

RMAN> connect target /
connected to target database: DB1020 (DBID=4124432604)

RMAN> connect auxiliary sys/oracle@oidrep.home.local
connected to auxiliary database: OIDREP (not mounted)

RMAN> duplicate target database to oidrep
2> pfile=D:\oracle\admin\oidrep\pfile\initoidrep.ora
3> db_file_name_convert=(
4>  'D:\oracle\oradata\db1020', 'D:\oracle\oradata\oidrep',
5>  'D:\oracle\oradata\db1020\rep', 'D:\oracle\oradata\oidrep\rep')
6> logfile 'D:\oracle\oradata\oidrep\redo01.log' size 100M,
7>  'D:\oracle\oradata\oidrep\redo02.log' size 100M,
8>  'D:\oracle\oradata\oidrep\redo03.log' size 100M;

Starting Duplicate Db at 03-JUN-07
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: sid=432 devtype=DISK

contents of Memory Script:
{
set newname for datafile  1 to
"D:\ORACLE\ORADATA\OIDREP\SYSTEM01.DBF";
set newname for datafile  2 to
[snip - this goes on and on]
"D:\ORACLE\ORADATA\OIDREP\REP\GDEFAULT1_OID.DBF";
set newname for datafile  27 to
"D:\ORACLE\ORADATA\OIDREP\REP\SVRMG1_OID.DBF";
restore
check readonly
clone database
;
}
executing Memory Script

executing command: SET NEWNAME
[snipped more of the same]
executing command: SET NEWNAME

Starting restore at 03-JUN-07
using channel ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: starting datafile backupset restore
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
restoring datafile 00001 to D:\ORACLE\ORADATA\OIDREP\SYSTEM01.DBF
restoring datafile 00002 to D:\ORACLE\ORADATA\OIDREP\UNDOTBS01.DBF
[snip - this goes on and on]
restoring datafile 00027 to D:\ORACLE\ORADATA\OIDREP\REP\SVRMG1_OID.DBF
channel ORA_AUX_DISK_1: reading from backup piece D:\ORACLE\DB\10.2.0\DATABASE\01IIVGIE_1_1
channel ORA_AUX_DISK_1: restored backup piece 1
piece handle=D:\ORACLE\DB\10.2.0\DATABASE\01IIVGIE_1_1 tag=TAG20070529T215523
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:09:37
Finished restore at 03-JUN-07
sql statement: CREATE CONTROLFILE REUSE SET DATABASE "OIDREP" RESETLOGS NOARCHIVELOG
MAXLOGFILES     16
MAXLOGMEMBERS      3
MAXDATAFILES      100
MAXINSTANCES     8
MAXLOGHISTORY      292
LOGFILE
GROUP  1 'D:\oracle\oradata\oidrep\redo01.log' SIZE 100 M ,
GROUP  2 'D:\oracle\oradata\oidrep\redo02.log' SIZE 100 M ,
GROUP  3 'D:\oracle\oradata\oidrep\redo03.log' SIZE 100 M
DATAFILE
'D:\ORACLE\ORADATA\OIDREP\SYSTEM01.DBF'
CHARACTER SET WE8MSWIN1252


contents of Memory Script:
{
switch clone datafile all;
}
executing Memory Script

released channel: ORA_AUX_DISK_1
datafile 2 switched to datafile copy
input datafile copy recid=1 stamp=624289967 filename=D:\ORACLE\ORADATA\OIDREP\UNDOTBS01.DBF
datafile 3 switched to datafile copy
input datafile copy recid=2 stamp=624289967 filename=D:\ORACLE\ORADATA\OIDREP\SYSAUX01.DBF
[snip - this goes on and on]
datafile 27 switched to datafile copy
input datafile copy recid=26 stamp=624289971 filename=D:\ORACLE\ORADATA\OIDREP\REP\SVRMG1_OID.DBF

contents of Memory Script:
{
recover
clone database
noredo
,
delete archivelog
;
}
executing Memory Script

Starting recover at 03-JUN-07
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: sid=431 devtype=DISK
Finished recover at 03-JUN-07

contents of Memory Script:
{
shutdown clone;
startup clone nomount pfile= 'D:\oracle\admin\oidrep\pfile\initoidrep.ora';
}
executing Memory Script

database dismounted
Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area     629145600 bytes

Fixed Size                     1292132 bytes
Variable Size                318769308 bytes
Database Buffers             301989888 bytes
Redo Buffers                   7094272 bytes
sql statement: CREATE CONTROLFILE REUSE SET DATABASE "OIDREP" RESETLOGS NOARCHIVELOG
MAXLOGFILES     16
MAXLOGMEMBERS      3
MAXDATAFILES      100
MAXINSTANCES     8
MAXLOGHISTORY      292
LOGFILE
GROUP  1 'D:\oracle\oradata\oidrep\redo01.log' SIZE 100 M ,
GROUP  2 'D:\oracle\oradata\oidrep\redo02.log' SIZE 100 M ,
GROUP  3 'D:\oracle\oradata\oidrep\redo03.log' SIZE 100 M
DATAFILE
'D:\ORACLE\ORADATA\OIDREP\SYSTEM01.DBF'
CHARACTER SET WE8MSWIN1252


contents of Memory Script:
{
set newname for tempfile  1 to
"D:\ORACLE\ORADATA\OIDREP\TEMP01.DBF";
switch clone tempfile all;
catalog clone datafilecopy  "D:\ORACLE\ORADATA\OIDREP\UNDOTBS01.DBF";
catalog clone datafilecopy  "D:\ORACLE\ORADATA\OIDREP\SYSAUX01.DBF";
[snip - this goes on and on]
catalog clone datafilecopy  "D:\ORACLE\ORADATA\OIDREP\REP\GDEFAULT1_OID.DBF";
catalog clone datafilecopy  "D:\ORACLE\ORADATA\OIDREP\REP\SVRMG1_OID.DBF";
switch clone datafile all;
}
executing Memory Script

executing command: SET NEWNAME

renamed temporary file 1 to D:\ORACLE\ORADATA\OIDREP\TEMP01.DBF in control file

cataloged datafile copy
datafile copy filename=D:\ORACLE\ORADATA\OIDREP\UNDOTBS01.DBF recid=1 stamp=624289989

cataloged datafile copy
datafile copy filename=D:\ORACLE\ORADATA\OIDREP\SYSAUX01.DBF recid=2 stamp=624289989

cataloged datafile copy
datafile copy filename=D:\ORACLE\ORADATA\OIDREP\USERS01.DBF recid=3 stamp=624289989
[snip - this goes on and on]
cataloged datafile copy
datafile copy filename=D:\ORACLE\ORADATA\OIDREP\REP\SVRMG1_OID.DBF recid=26 stamp=624289994

datafile 2 switched to datafile copy
input datafile copy recid=1 stamp=624289989 filename=D:\ORACLE\ORADATA\OIDREP\UNDOTBS01.DBF
datafile 3 switched to datafile copy
input datafile copy recid=2 stamp=624289989 filename=D:\ORACLE\ORADATA\OIDREP\SYSAUX01.DBF
datafile 4 switched to datafile copy
[snip - this goes on and on]
datafile 27 switched to datafile copy
input datafile copy recid=26 stamp=624289994 filename=D:\ORACLE\ORADATA\OIDREP\REP\SVRMG1_OID.DBF

contents of Memory Script:
{
Alter clone database open resetlogs;
}
executing Memory Script

database opened
Finished Duplicate Db at 03-JUN-07

RMAN> exit
Recovery Manager complete.

C:\Documents and Settings\frankbo>time /t
01:54 PM

So - all in all the cloning took 13 minutes.
In contrast to earlier releases, that did not create the tempfile, belonging to the temporary tablespace, there is no more need to create a tempfile - it's there!
C:\Documents and Settings\frankbo>set oracle_sid=oidrep
C:\Documents and Settings\frankbo>sqlplus / as sysdba

SQL*Plus: Release 10.2.0.3.0 - Production on Sun Jun 3 14:02:11 2007
Copyright (c) 1982, 2006, Oracle.  All Rights Reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Production
With the Partitioning, OLAP and Data Mining options

SQL> select name from v$tempfile;
NAME
--------------------------------------------------------------------------------
D:\ORACLE\ORADATA\OIDREP\TEMP01.DBF

SQL> create spfile from pfile='D:\oracle\admin\oidrep\pfile\initoidrep.ora';
File created.

Bounce the db, to make sure the spfile is picked up
SQL> show parameter pfile

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      D:\ORACLE\DB\10.2.0\DATABASE\SPFILEOIDREP.ORA
SQL> select dbid, db_unique_name from v$database;

DBID       DB_UNIQUE_NAME
---------- ------------------------------
3574270531 oidrep

SQL> connect sys/manager@db1020.home.local as sysdba
Connected.
SQL> /

DBID DB_UNIQUE_NAME
---------- ------------------------------
4124432604 db1020

No worries about dbid, either...

That concludes the database preparations.

Installation phase 4: prepare the Network
I made a distinction between two stages: getting the balancer, and adding virtual addresses.
First off, a little bit about the setup. As said earlier on, I (only) have 3 machines, and the complete configuration requires ate least four, better yet, six. As 6=3*2, every machine gets a double function, some even triple functions (and no - you do not want to VMWare this - your host will not cope with it...)

Phase 4a: get, make and install balance.
Logon to your machines as root. I need the c-compiler, so let's get it:
yum install gcc

Next, download the source tarball from http://www.inlab.de/balance-3.35.tar.gz to /install
[root@idmhost ~]# cd /install/
[root@idmhost install]# gunzip balance-3.35.tar.gz
[root@idmhost install]# tar -xf balance-3.35.tar
[root@idmhost install]# cd balance-3.35

Now, if you would run make make install at this stage, you would get a (minor) error; there's a slight typographical error on line 11 of the makefile, so change the Makefile file:
#MANDIR=${BINDIR}/../man/man1
MANDIR=/usr/share/man/man1

And run "make install":
[root@idmhost balance-3.35]# make install
install -o root -g root -m 755  balance \
/usr/sbin/balance
install -o root -g root -m 755  balance.1 \
/usr/share/man/man1
mkdir -p /var/run/balance
chmod 1777 /var/run/balance
[root@idmhost balance-3.35]#

Not doing so, will lead to this error:
[root@idmhost balance-3.35]# make install
install -o root -g root -m 755  balance \
/usr/sbin/balance
install -o root -g root -m 755  balance.1 \
/usr/sbin/../man/man1
install: cannot create regular file `/usr/sbin/../man/man1': No such file or directory
make: *** [install] Error 1
[root@idmhost balance-3.35]#

No harm done, simply edit the Makefile, and rerun... Failure to do so will not have any effect on the program, you just will not have the man-pages.

Phase 4b: virtual addresses and names to your local network.
Remember, basically, I wanted:
  • A load-balanced request to two SSO servers.
  • Those SSO servers request a loadbalanced OID.
  • Those two OID processes use SQL*Net time out and loadbalancing to query two active databases, which are clones of eachother.
Remember also, I only have three machines.

So, network wise, I would need:
  • Two SSO instances (addresses: IDM_IP1 and IDM_IP2), being served by a loadbalancer. This loadbalancer is actually a HTTP balancer, serving the SSO and DAS pages (the Identity Management Layer).
    As you do not want to bother people with the distinction between IDM_IP1 or IDM_IP2, the balancer should have a name. From now on, that is login.home.local. Ip address is IDM_IP0.
  • The SSO/DAS pages are requesing OID services through a load balancer, but that is an LDAP loadbalancer. It only needs to server LDAP requests (I am going to use the non-priveleged port range, 3060 (non-ssl) and 3130 (SSL), in stead of the default 386 and 636).
Not sure if that balancer should have a name; it's address ip OID_OP0, and it balances between OID_IP1 and OID_IP2.

Physically, I have used the machine names OIDHOST and IDMHOST so far. See previous posting about that. What I'm going to do, is install the first OID and IDM installs on the OIDHOST and IDMHOST repectively, and the second OID and IDM installs go on the IDMHOST and OIDHOST respectively.
Both application servers will serve OID as well as IDM:
IDMHOST:    IDM1    OID2
OIDHOST:    IDM2    OID1
The first loadbalancer, the HTTP one, will sit on IDMHOST, the second will sit on OIDHOST.
This leads to:
IDMHOST/original address:   192.168.1.220
IDMHOST/IDM1 address:       192.168.1.225
IDMHOST/OID2 address:       192.168.1.226
IDMHOST/login.home.local:   192.168.1.227
OIDHOST/original address:   192.168.1.210
OIDHOST/IDM2 address:       192.168.1.215
OIDHOST/OID1 address:       192.168.1.216
OIDHOST/ldapbal.home.local: 192.168.1.217

In addition: db1020.home.local resides on 192.128.1.104, as does oidrep, the replication instance.

Let's add the addresses:
On IDMHOST:
ifconfig eth0:1 192.168.1.225
ifconfig eth0:2 192.168.1.226
ifconfig eth0:3 192.168.1.227

On OIDHOST:
ifconfig eth0:1 192.168.1.215
ifconfig eth0:2 192.168.1.216
ifconfig eth0:3 192.168.1.217

Check by running ifconfig:
eth0      Link encap:Ethernet  HWaddr 00:50:DA:4A:BC:2A
inet addr:192.168.1.210  Bcast:192.168.1.255  Mask:255.255.255.0
inet6 addr: fe80::250:daff:fe4a:bc2a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:12641 errors:0 dropped:0 overruns:1 frame:0
TX packets:9048 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:9696620 (9.2 MiB)  TX bytes:1106121 (1.0 MiB)
Interrupt:169 Base address:0xd800

eth0:1    Link encap:Ethernet  HWaddr 00:50:DA:4A:BC:2A
inet addr:192.168.1.215  Bcast:192.168.1.255  Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
Interrupt:169 Base address:0xd800

eth0:2    Link encap:Ethernet  HWaddr 00:50:DA:4A:BC:2A
inet addr:192.168.1.216  Bcast:192.168.1.255  Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
Interrupt:169 Base address:0xd800

eth0:3    Link encap:Ethernet  HWaddr 00:50:DA:4A:BC:2A
inet addr:192.168.1.217  Bcast:192.168.1.255  Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
Interrupt:169 Base address:0xd800
Or use Webmin, the Networking entry, Network Configuration, Network Interfaces.
You may alse define (permanent) virtual addresses here. If you insist on doing it by hand, create the appropiate files (ifcfg-eth0:1, etc) in /etc/sysconfig/network-scripts:
BOOTPROTO=none
DEVICE=eth0:1
NETMASK=255.255.255.0
MTU=1500
BROADCAST=192.168.1.255
ONPARENT=yes
IPADDR=192.168.1.225
NETWORK=192.168.1.0
ONBOOT=yes

Alternatively, add the ifconfig eth0:1 lines to /etc/rc.local:

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local
/sbin/ifconfig eth0:1 192.168.1.215
/sbin/ifconfig eth0:2 192.168.1.216
/sbin/ifconfig eth0:3 192.168.1.217

Change the hosts files on all machines, under Linux, it is /etc/hosts:
127.0.0.1               localhost.localdomain localhost
192.168.1.210           oidhost.home.local
192.168.1.220           idmhost.home.local
192.168.1.104           dbhost.home.local
192.168.1.225           idm1.home.local idm1
192.168.1.226           oid2.home.local oid2
192.168.1.227           login.home.local login
192.168.1.215           idm2.home.local idm2
192.168.1.216           oid1.home.local oid1
192.168.1.217           ldapbalancer.home.local ldapbalancer

Do not foget to add these to the database host (C:\WINDOWS\system32\drivers\etc)! Failing in doing so will reslove in nasty install errors
(ORA-31203: DBMS_LDAP: PL/SQL - Init Failed, java class not found)

Let's reboot the systems to see if everything acts as we want: reboot -n
Try to ping every host defined, from every machine. If that is successfull, let's do the vandango:
[root@idmhost ~]# balance -b login.home.local http idm1:http % idm2:http %
[root@idmhost ~]# balance -b login.home.local https idm1:https % idm2:https %

Similar:
[root@oidhost ~]# balance -b ldapbalancer.home.local 3060 oid1:3060 oid2:3060
[root@oidhost ~]# balance -b ldapbalancer.home.local 3130 oid1:3130 oid2:3130

That concludes phase 4.
Installation phase 5: Oracle Internet DirectoryPhase 5a: Preliminaries.
On both machines, create distinct groups and user:

[root@oidhost ~]# groupadd oidown
[root@oidhost ~]# groupadd oidinst
[root@oidhost ~]# useradd oidoracle -g oidinst -G oidown -c 'Oracle Internet Directory software owner'
[root@oidhost ~]# passwd oidoracle
Changing password for user oidoracle.
New UNIX password:
Retype new UNIX password:
passwd: all authentication tokens updated successfully.

Make sure I can unpack the cpio and zipped files in the /install directory (which is not owned by oidoracle!)

[root@oidhost ~]# chmod 777 /install

Create the installation directory, and change ownership:

[root@oidhost ~]# mkdir -p /oracle/ias/oraInventory
[root@oidhost ~]# chown -R oidoracle:oidown /oracle
[root@oidhost ~]# su - oidoracle
[oidoracle@oidhost ~]$ cd /install
[oidoracle@oidhost install]$ cpio -idmv< /install/as_linux_x86_portal_wireless_101202_disk1.cpio [oidoracle@oidhost install]$ cpio -idmv< /install/as_linux_x86_portal_wireless_101202_disk2.cpio [oidoracle@oidhost install]$ cpio -idmv< /install/as_linux_x86_portal_wireless_101202_disk3.cpio [oidoracle@oidhost install]$ cpio -idmv< /install/as_linux_x86_portal_wireless_101202_disk4.cpio [oidoracle@oidhost install]$ unzip p4960210_10122_LINUX.zip -d p4960210 [oidoracle@oidhost install]$ unzip p5901894_10122_LINUX.zip -d p5901894 [oidoracle@oidhost install]$ unzip p5922121_10122_LINUX.zip -d p5922121

Phase 5a: first OID install.
I was planning on using non-default ports, so let's do some prepartion for that:
[oidoracle@oidhost install]$ cp Disk1/stage/Response/staticports.ini /oracle/ias/staticports.ini
Now, I need to make the installer aware of the fact, I want ports 3060 and 3130 to be used. The interesting part of staticports.ini:
# Infrastructure

Oracle Internet Directory port = 3060
Oracle Internet Directory (SSL) port = 3130
#Oracle Certificate Authority SSL Server Authentication port = port_num
#Oracle Certificate Authority SSL Mutual Authentication port = port_num
#Ultra Search HTTP port number = port_num

OK - let's fire up the installer:
[oidoracle@oidhost ~]$ export DISPLAY=192.168.1.104:0.0
[oidoracle@oidhost ~]$ /install/Disk1/runInstaller

Enter the correct locations, and...

Let's do what is asked for...

Once more - correct locations...

Infrastucture install...


Let's do Identity Management.


Yeah - don't feel like upping it even further - besides, these are the values, specified in the Linux Installation Manual... Just mark them as okay, and continue.
Of course we have root priveleges - I am not really going to upload a picture showing how to confirm that, just continue to the next:


Remember the envisioned setup: The LDAP services (OID) and Integration will be running here, and the rest (SSO and DAS, no CA this time) on the Identity Management Host (idmhost.home.local)

With all the preparations, make sure we use them! Select the correct file.



Ebter the correct data, and...

What the ...?!? The Oracle Application Server Metadata Repository is not compatible?!? I checked and doublechecked versions - no error there! Back to the drawingboard!

Update:
As far as I can tell, Metalink came up empty, Google came up empty and so did tahiti. I admit, I did not look at all references matching my search criteria, because a lot of hits are about backwards compatibility problems. And I know for a fact, the MCRA versions 10.1.2.0.0 and 10.1.2.0.2 are incompatble, too.
The screen itself leaves no room for informative queries, so all that is left is the log file of the installation itself. This looks like:
Calling Query DBConnectQueries8.2  GetSchemaVer

SchemaName = *Protected value, not to be logged*

SchemaPassword = *Protected value, not to be logged*

ConnectString = 192.168.1.104:1521/db1020.home.local

SqlQuery = select attrval from ods.ds_attrstore where entryid=1 and attrname = 'orcldirectoryversion'
Query Returned: OID 10.1.2.1.0
OID Schema value returned from SQL is OID 10.1.2.1.0.

Extracted version is 10.1.2.1.0.
Calling Query DBConnectQueries8.2  IsOIDConfigured

SchemaName = *Protected value, not to be logged*

SchemaPassword = *Protected value, not to be logged*

ConnectString = 192.168.1.104:1521/db1020.home.local
Query Returned: false
Calling Query DBConnectQueries8.2  IsUserWithDBAPriv

User = *Protected value, not to be logged*

Password = *Protected value, not to be logged*

ConnectString = 192.168.1.104:1521/db1020.home.local
Query Returned: true
Calling Query DBConnectQueries8.2  GetRepositoryVer

User = *Protected value, not to be logged*

Password = *Protected value, not to be logged*

ConnectString = 192.168.1.104:1521/db1020.home.local
Query Returned: Null
Using the default value for query.
Error:*** Alert: The Oracle Application Server Metadata Repository that you have specified is not a compatible version for configuring Oracle Internet Directory. Please specify another database. ***

What I understand from this, is the fact that OID is not configured, causes the installer to abort. Of course OID isn't configured - I choose to install that!
Anyway - somewhere deep (in /install/Disk1/stage/Queries/DBConnectQueries/8.2/1) there is a file, called DBConnectQueries.jar. Opening it, and searching for GetRepositoryVer showed some interesting stuff (like the development machine, syndey.oracle.com, with system password!), like:
select version from app_registry where comp_id = 'MRC';
select version from ias_versions where id = 'mrc';

I cannot tell where the second query comes in, but the first does resolve:
SQL> select comp_id, version, status from app_registry;

COMP_ID                        VERSION                        STATUS
------------------------------ ------------------------------ -----------
PORTAL                         10.1.2.0.2                     VALID
SSO                            10.1.2.0.2                     VALID
WORKFLOW                       10.1.2.0.2                     VALID
B2B                            10.1.2.0.2                     VALID
BAM                            10.1.2.0.2                     VALID
MRC                                                           LOADING
OCA                            10.1.2.0.2                     VALID
OID                            10.1.2.0.2                     VALID
DCM                            10.1.2.0.2                     VALID
DISCOVERER                     10.1.2.0.2                     VALID


I fired up the MRCA again, and tried to redo the install. Nope - remove first, and only then install... Remove drops objects, before dropping tablespaces. There is a faster way to do that... had to do it twice, no indication why, the last line of the first sessions' log reads:
Repository Loader actionStarting
The correct, completed session goes on after that:
Repository Loader actionStarting
Repository Loader actionFinished
Repository Loader ActionQueueFinished
Unloading...
And continues dropping tablespaces, and explaining the wizard has stopped, about twenty times. Mysteries...

During the process, I observed:
SQL> select comp_id, version, status from app_registry;
no rows selected

SQL> /

COMP_ID                        VERSION                        STATUS
------------------------------ ------------------------------ -----------
MRC                                                           LOADING
DISCOVERER                     10.1.2.0.2                     VALID
DCM                            10.1.2.0.2                     VALID

SQL> /

COMP_ID                        VERSION                        STATUS
------------------------------ ------------------------------ -----------
PORTAL                                                        LOADING
SSO                            10.1.2.0.2                     VALID
WORKFLOW                       10.1.2.0.2                     VALID
B2B                            10.1.2.0.2                     VALID
BAM                            10.1.2.0.2                     VALID
MRC                                                           LOADING
OCA                            10.1.2.0.2                     VALID
OID                            10.1.2.0.2                     VALID
DISCOVERER                     10.1.2.0.2                     VALID
DCM                            10.1.2.0.2                     VALID

10 rows selected.
SQL> /

COMP_ID                        VERSION                        STATUS
------------------------------ ------------------------------ -------
SYNDICATION                    10.1.2.0.2                     VALID
PORTAL                         10.1.2.0.2                     VALID
SSO                            10.1.2.0.2                     VALID
WORKFLOW                       10.1.2.0.2                     VALID
B2B                            10.1.2.0.2                     VALID
BAM                            10.1.2.0.2                     VALID
MRC                            10.1.2.0.2                     VALID
OCA                            10.1.2.0.2                     VALID
OID                            10.1.2.0.2                     VALID
WIRELESS                       10.1.2.0.2                     VALID
DISCOVERER                     10.1.2.0.2                     VALID
DCM                            10.1.2.0.2                     VALID
WCS                            10.1.2.0.2                     VALID
UDDI                           10.1.2.0.2                     VALID


That seems to be different from where I started - but the MCRA did finish OK...
Well, back to cloning and then retry the install!

Update:
Started the machines, database instance and listener, balancer om both machines.
Checked hosts. Installer continued smoothly this time:

I left it for what it was - you may consider otherwise, especially when you have plans on extending the root entry (.local, in this case). For .com it may not be such a problem, but for .nl it will be - imagine your company extends abroad. In that case, consider a megalomaniac '.world' as root: your.company.nl.world can expand into your.other.be.world.

MDS stands for Master Definition Site...

229 products(!) to be installed. And I did not even select all options!

Let's take a closer look at the log, then:
Leaving Ldap Post Installation Set File Permissions
Stopping  OID Server using OPMN..
Starting OID Server using OPMN..
Mon Jun 18 19:09:49 CEST 2007 Bind request issued. Waiting for OID Server response.
with a retryCount:20
Mon Jun 18 19:10:19 CEST 2007 Bind request issued. Waiting for OID Server response.
javax.naming.CommunicationException: oidhost.home.local:3060 [Root exception is java.net.ConnectException: Connection refused]

OK - see if the process actually runs; switch to $ORACLE_HOME/opmn/bin, and:
[oidoracle@oidhost bin]$ ./opmnctl status

Processes in Instance: mds.oidhost.home.local
-------------------+--------------------+---------+---------
ias-component      | process-type       |     pid | status
-------------------+--------------------+---------+---------
DSA                | DSA                |     N/A | Down
LogLoader          | logloaderd         |     N/A | Down
dcm-daemon         | dcm-daemon         |    5816 | Alive
HTTP_Server        | HTTP_Server        |     N/A | Down
OID                | OID                |     N/A | Down
No wonder, OID is down... Let's just start all processes:
[oidoracle@oidhost bin]$ ./opmnctl startall
opmnctl: starting opmn and all managed processes...
[oidoracle@oidhost bin]$ ./opmnctl status

Processes in Instance: mds.oidhost.home.local
-------------------+--------------------+---------+---------
ias-component      | process-type       |     pid | status
-------------------+--------------------+---------+---------
DSA                | DSA                |     N/A | Down
LogLoader          | logloaderd         |     N/A | Down
dcm-daemon         | dcm-daemon         |    5816 | Alive
HTTP_Server        | HTTP_Server        |    7753 | Alive
OID                | OID                |    7758 | Alive

Still - retry fails. Then I realize, I already switched on loadbalancing... and sure enough, after killing these balance processes, the wizards continued, only to fail once more:

This is a bit of a silly error message: opmn cannot start the process, because I already started it! Resolution: stop the process manually:
[oidoracle@oidhost bin]$ ./opmnctl stopproc type=ohs
opmnctl: stopping opmn managed processes...

Some (actually, a lot) of wizards later, this is the reward:


Update: (Phase 5c-second OID install)Started both instances, and opened the databases. Logged on to oidhost, and changed .bash_profile; added those lines:
export ORACLE_HOME=/oracle/ias/oid10.1.2
export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/opmn/bin:$PATH

That allows me to:
[oidoracle@oidhost ~]$ opmnctl startall
opmnctl: starting opmn and all managed processes...
[oidoracle@oidhost ~]$ opmnctl status

Processes in Instance: mds.oidhost.home.local
-------------------+--------------------+---------+---------
ias-component      | process-type       |     pid | status
-------------------+--------------------+---------+---------
DSA                | DSA                |     N/A | Down
LogLoader          | logloaderd         |     N/A | Down
dcm-daemon         | dcm-daemon         |    3906 | Init
HTTP_Server        | HTTP_Server        |    3904 | Alive
OID                | OID                |    3912 | Alive

Logged on to the idmhost, with oidoracle account. Editied the localhosts file again, with the following contents:
Oracle Internet Directory port = 3060
Oracle Internet Directory (SSL) port = 3130
#Oracle Certificate Authority SSL Server Authentication port = port_num
#Oracle Certificate Authority SSL Mutual Authentication port = port_num
#Ultra Search HTTP port number = port_num

Fired up the installer:
[oidoracle@idmhost oracle]$ /install/Disk1/runInstaller -paramFile /oracle/ias/oraparam.ini

Only screens that do differ from above are loaded:
Select three options: Internet Directory, Directory Integration and HA/Replication.


Indicate the correct location of the staticports.ini file.
I had to use SYSTEM here - could not get SYS to work:

Hmmmmmm.... I don't want to choose here! I want both. Maybe this is the reason clustered installs don't replicate? In this manner, there are two farms, and farms cannot cluster. Only whatever application server instance belongs to the farm, can participate in a cluster: 1 farm == 1 repository.
Maybe when I base the instance on a file-based repository, on a shared disk?!?

Next screen, select Replication:


Next screen, select Advanced Replication.


Now, this one is tricky: it states "Master Node", where in fact, this is the second install. True, but this is Multi Master Replication, so in fact: there are no masters (or everyone is the master)!


Same here: "Master", but watch out: the data entered actually refers to the real master, the first installed instance: oidhost.home.local!


Provide the correct connection information, and get used to the "cn=" notation - this is LDAP land... Note the naming of the instance: rms, as in "Replicated Master Site".


That's it... the installer will install, the wizzards wizz, and it all ends in:


Update: Something went wrong, I noticed after reflection. I miss one installer screen; the one that allows me to select the (virtual) ip address and (virtual) server name! It should have been presented because of the changes I made to oraparam.ini (SHOW_HOSTNAME=ALWAYS_SHOW) .

Update:
Before attempting to get replication to work, I'll need to fix the network component. That means adding the "other" entry to each tnsnames.ora, so each file is identical:
OIDREP.home.local =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = dbhost.home.local)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = oidrep.home.local)
)
)

db1020.home.local =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = dbhost.home.local)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = db1020.home.local)
)

It also means, I need to add a default domain - OID seems to make it a habit of sometimes using a domain qualified call, sometimes not. Consequently, db1020 as well as db1020.home.local must be resolved. Added this to sqlnet.ora:
names.default_domain=home.local

The same is true for the database server(s); they need to be able to connect lateron - afterall, it is database based replication, not Application Server!

Next stop: replication!

Update: (Phase 6 - install Replication)
After all these preparations, starting replication should be quite easy: use the remtool (reminding me of a REMoval tool, what's in a name?): (some logging has been snipped to save space)
[oidoracle@oidhost oid10.1.2]$ remtool -asrsetup -v
------------------------------------------------------------------------------
ASR Setup for OID Replication
WARNING:
Make sure that the replication administrator that you
enter below does not exist already in any of the nodes
that will be part of the DRG to be created now. If the
user exists, that user will be dropped and will be
created newly.
------------------------------------------------------------------------------
Enter replication administrator's name       : repadmin

Enter replication administrator's password   :
Reenter replication administrator's password :
Enter Master Definition Site (MDS) details   :
Enter global name of MDS                     : db1020.home.local

Enter SYSTEM user password of MDS            :
Enter Remote Master Site (RMS) details       :
Enter global name of RMS #  1                : oidrep.home.local

Enter SYSTEM user password of RMS #  1       :
Are there more Remote Master Sites in the group? [y/n/q] : n

Verify the details you had entered.
------------------------------------------------------------------------------
Replication administrator's name   : repadmin
Master Definition Site             : db1020.home.local
Remote Master Site #  1            : oidrep.home.local
Are these details correct? [y/n/q] : y

------------------------------------------------------------------------------
ASR setup in progress...

DB1020.HOME.LOCAL : Verifying uniqueness of replication agreement entry...
DB1020.HOME.LOCAL : Dropping replication administrator repadmin...
DB1020.HOME.LOCAL : Creating replication administrator repadmin...
DB1020.HOME.LOCAL : Granting privileges or roles required for replication administrator to repadmin...
DB1020.HOME.LOCAL : Granting privileges or roles required for replication administrator to repadmin...
DB1020.HOME.LOCAL : Granting privileges or roles required for replication administrator to repadmin...
DB1020.HOME.LOCAL : Creating purge job...
DB1020.HOME.LOCAL : Dropping database link made to OIDREP.HOME.LOCAL...
DB1020.HOME.LOCAL : Dropping database link made to OIDREP.HOME.LOCAL...
DB1020.HOME.LOCAL : Creating database link to OIDREP.HOME.LOCAL...
DB1020.HOME.LOCAL : Scheduling push job to OIDREP.HOME.LOCAL...
OIDREP.HOME.LOCAL : Verifying uniqueness of replication agreement entry...
OIDREP.HOME.LOCAL : Dropping replication administrator repadmin...
OIDREP.HOME.LOCAL : Creating replication administrator repadmin...
OIDREP.HOME.LOCAL : Granting privileges or roles required for replication administrator to repadmin...
OIDREP.HOME.LOCAL : Granting privileges or roles required for replication administrator to repadmin...
OIDREP.HOME.LOCAL : Granting privileges or roles required for replication administrator to                       repadmin...
OIDREP.HOME.LOCAL : Creating purge job...
OIDREP.HOME.LOCAL : Dropping database link made to DB1020.HOME.LOCAL...
OIDREP.HOME.LOCAL : Creating database link to DB1020.HOME.LOCAL...
OIDREP.HOME.LOCAL : Scheduling push job to DB1020.HOME.LOCAL...
DB1020.HOME.LOCAL : Dropping replication group LDAP_REP...
DB1020.HOME.LOCAL : Creating replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ODS.ASR_CHG_LOG to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ODS.ODS_CHG_STAT to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSSO_LS_CONFIGURATION_INFO_T to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSSO_PS_CONFIGURATION_INFO_T to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSSO_PAPP_CONFIGURATION_INF_T to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSSO_PSEX_APP_INFO$ to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSSO_PSEX_USER_INFO$ to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSSO_ANNOUNCEMENT_CONFIG_T to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWHOSTING_SWITCH$ to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSEC_PERSON$ to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWCTX_COOKIE_INFO$ to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSSO_APPLICATION_INFO_T to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSSO_APPUSERINFO_T to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSEC_ENABLER_CONFIG_INFO$ to replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding object TABLE ORASSO.WWSUB_MODEL$ to replication group LDAP_REP...
OIDREP.HOME.LOCAL : Dropping replication group LDAP_REP...
DB1020.HOME.LOCAL : Adding replication site OIDREP.HOME.LOCAL to replication group LDAP_REP...
DB1020.HOME.LOCAL : Executing deferred administrative requests...
OIDREP.HOME.LOCAL : Executing deferred administrative requests...
DB1020.HOME.LOCAL : Generating replication support for TABLE ODS.ASR_CHG_LOG...
DB1020.HOME.LOCAL : Executing deferred administrative requests...
OIDREP.HOME.LOCAL : Executing deferred administrative requests...
DB1020.HOME.LOCAL : Generating replication support for TABLE ODS.ODS_CHG_STAT...
DB1020.HOME.LOCAL : Executing deferred administrative requests...
OIDREP.HOME.LOCAL : Executing deferred administrative requests...
DB1020.HOME.LOCAL : Generating replication support for TABLE ORASSO.WWSSO_LS_CONFIGURATION_INFO_T...
ORASSO.WWSSO_PS_CONFIGURATION_INFO_T...
ORASSO.WWSSO_PAPP_CONFIGURATION_INF_T...
ORASSO.WWSSO_PSEX_APP_INFO$...
ORASSO.WWSSO_PSEX_USER_INFO$...
ORASSO.WWSSO_ANNOUNCEMENT_CONFIG_T...
ORASSO.WWHOSTING_SWITCH$...
ORASSO.WWSEC_PERSON$...
ORASSO.WWCTX_COOKIE_INFO$...
ORASSO.WWSSO_APPLICATION_INFO_T...
ORASSO.WWSSO_APPUSERINFO_T...
ORASSO.WWSEC_ENABLER_CONFIG_INFO$...
ORASSO.WWSUB_MODEL$...
DB1020.HOME.LOCAL : Verifying initialization parameter...
DB1020.HOME.LOCAL : Altering init param value of global_names to TRUE...
CORRECTED:
DB1020.HOME.LOCAL : Initialization parameter global_names' value has been altered to TRUE.                       Alter INIT.ORA file to reflect the above change.
OIDREP.HOME.LOCAL : Verifying initialization parameter...
OIDREP.HOME.LOCAL : Altering init param value of global_names to TRUE...
CORRECTED:
OIDREP.HOME.LOCAL : Initialization parameter global_names' value has been altered to TRUE.                       Alter INIT.ORA file to reflect the above change.
DB1020.HOME.LOCAL : Verifying uniqueness of replication agreement entry...
OIDREP.HOME.LOCAL : Verifying uniqueness of replication agreement entry...
DB1020.HOME.LOCAL : Verifying replication agreement entry...
DB1020.HOME.LOCAL : Inserting replication agreement entry oidhost_db1020...
CORRECTED:
DB1020.HOME.LOCAL : "oidhost_db1020" hostname has been added to replication agreement entry.
DB1020.HOME.LOCAL : Inserting replication agreement entry idmhost_oidrep...
CORRECTED:
DB1020.HOME.LOCAL : "idmhost_oidrep" hostname has been added to replication agreement entry.
OIDREP.HOME.LOCAL : Verifying replication agreement entry...
OIDREP.HOME.LOCAL : Inserting replication agreement entry oidhost_db1020...
CORRECTED:
OIDREP.HOME.LOCAL : "oidhost_db1020" hostname has been added to replication agreement entry.
OIDREP.HOME.LOCAL : Inserting replication agreement entry idmhost_oidrep...
CORRECTED:
OIDREP.HOME.LOCAL : "idmhost_oidrep" hostname has been added to replication agreement entry.
DB1020.HOME.LOCAL : Resuming replication activity...
DB1020.HOME.LOCAL : Executing deferred administrative requests...
OIDREP.HOME.LOCAL : Executing deferred administrative requests...
------------------------------------------------------------------------------
ASR setup has been configured successfully.
------------------------------------------------------------------------------
Directory Replication Group (DRG) details :

-------- ------------- ----------------------- ------------- ------------- ----
Instance Host Name     Global Name             Version       Replicaid     Site
Name                                                                       Type
-------- ------------- ----------------------- ------------- ------------- ----
db1020   CS-FRANK03    DB1020.HOME.LOCAL       OID 10.1.2.1. oidhost_db102 MDS
oidrep   CS-FRANK03    OIDREP.HOME.LOCAL       OID 10.1.2.1. idmhost_oidre RMS
-------- ------------- ----------------------- ------------- ------------- ----

[oidoracle@oidhost oid10.1.2]$


If the setup fails with
ORA-12154: TNS:could not resolve the connect identifier specified
in the dropping/creating database links part, right at the beginning, make sure global_name (select * from global_name) is the same as your service_name in tnsnames.ora.

Now, start replication services, and see if they run:
[oidoracle@oidhost oid10.1.2]$ oidctl connect=db1020.home.local server=oidrepld instance=1 flags="-h oidhost.home.local -p 3060" start
[oidoracle@oidhost oid10.1.2]$ $ORACLE_HOME/ldap/bin/ldapcheck

Checking Oracle Internet Directory Processes ...ALL

Process oidmon is Alive as PID 3897
Process oidldapd is Alive as PID 3898
Process oidldapd is Alive as PID 3904
Process oidrepld is Alive as PID 8451
Process odisrv is Alive as PID 3899

Same thing on other machine:
[oidoracle@idmhost bin]$ oidctl connect=oidrep.home.local server=oidrepld instance=1 flags="-h idmhost.home.local -p 389" start
Waiting for OIDMON to stop OIDREPLD, see oidmon.log for details.
[oidoracle@idmhost bin]$ ./ldapcheck

Checking Oracle Internet Directory Processes ...ALL

Process oidmon is Alive as PID 3602
Process oidldapd is Alive as PID 3611
Process oidldapd is Alive as PID 3615
Process oidrepld is Alive as PID 5457
Process odisrv is Alive as PID 3612
Does it work?
Well, fire up the Directory Manager, connect to both LDAP servers, and navigate to cn=Entry Management,dc=local,dc=home,cn=users,cn=orcladmin.
On the first machine, oidhost, you will see this (notice the timestamp):


The replicated machine, idmhost, will show this:

Note, not only are the timestamps the same, and I did not do the two installs simultaniously, but the modifiersname is the replication process:
cn=replication dn,orclreplicaid=idmhost_oidrep,cn=replication configuration
Next step: install the Single Sign On and Delegated Administration Services

Update:

Starting up all processes (e.g. after a startup; I do not leave my test machines on 24*7), is as easy as 1-2-3:
Last login: Fri Jun 22 08:30:59 2007 from dbhost.home.local
[oidoracle@idmhost ~]$ opmnctl startall
opmnctl: starting opmn and all managed processes...
[oidoracle@idmhost ~]$ opmnctl status

Processes in Instance: rms.idmhost.home.local
-------------------+--------------------+---------+---------
ias-component      | process-type       |     pid | status
-------------------+--------------------+---------+---------
DSA                | DSA                |     N/A | Down
LogLoader          | logloaderd         |     N/A | Down
dcm-daemon         | dcm-daemon         |     N/A | Down
HTTP_Server        | HTTP_Server        |    3503 | Alive
OID                | OID                |    3518 | Alive

[oidoracle@idmhost ~]$ $ORACLE_HOME/ldap/bin/ldapcheck

Checking Oracle Internet Directory Processes ...ALL

Process oidmon is Alive as PID 3518
Process oidldapd is Alive as PID 3531
Process oidldapd is Alive as PID 3537
Process oidrepld is Alive as PID 3565
Not Running ---- Process odisrv

This odisrv is a bit of a nag. It is running perfectly on the other machine:
[oidoracle@oidhost ~]$ $ORACLE_HOME/ldap/bin/ldapcheck

Checking Oracle Internet Directory Processes ...ALL

Process oidmon is Alive as PID 3944
Process oidldapd is Alive as PID 3978
Process oidldapd is Alive as PID 3981
Process oidrepld is Alive as PID 4013
Process odisrv is Alive as PID 3983

However, opmnctl does not seem to control it, after a few stopall and startall, I had this:
[oidoracle@oidhost ~]$ $ORACLE_HOME/ldap/bin/ldapcheck

Checking Oracle Internet Directory Processes ...ALL

Process oidmon is Alive as PID 6410
Process oidldapd is Alive as PID 6411
Process oidldapd is Alive as PID 6426
Process oidrepld is Alive as PID 6585
Process odisrv is Alive as PID 6170
Process odisrv is Alive as PID 6414

Oh well. What bothers me is the fact odisrv does not run on idmhost; the log shows:
-----------------------------------------------------
Oracle Directory Integration Server instance# 01 started..
-----------------------------------------------------
Sat Jun 23 12:59:08 CEST 2007 : Starting Server to execute Profile Group :default against LDAP Server (idmhost.home.local:3130)
Sat Jun 23 12:59:09 CEST 2007 : SSL Mode :1
Sat Jun 23 12:59:09 CEST 2007 : Exception :javax.naming.AuthenticationException: [LDAP: error code 49 - Invalid Credentials]
Sat Jun 23 12:59:09 CEST 2007 : Aborting.. : null
Sat Jun 23 12:59:09 CEST 2007 : Exiting with Status -1: null

On odihost, the correct startup message in the log:
-----------------------------------------------------
Oracle Directory Integration Server instance# 01 started..
-----------------------------------------------------
Sat Jun 23 12:26:56 CEST 2007 : Starting Server to execute Profile Group :default against LDAP Server (oidhost.home.local:3130)
Sat Jun 23 12:26:56 CEST 2007 : SSL Mode :1
Guess I need to sort that out, before continuing to the next step.

Update: (don't try this - see below)
Change the port on idmhost.home.local from 389 to 3060, ran dcmctl updateconfig.
Then, I ran this, and all of a sudden, it worked!

[oidoracle@idmhost log]$ odisrvreg -D cn=orcladmin -w Welcome1 -p 3060
Registering for the first time...
DIS registration successful.

[oidoracle@idmhost log]$ $ORACLE_HOME/ldap/bin/ldapcheck

Checking Oracle Internet Directory Processes ...ALL

Process oidmon is Alive as PID 5645
Process oidldapd is Alive as PID 5648
Process oidldapd is Alive as PID 5660
Process oidrepld is Alive as PID 5697
Process odisrv is Alive as PID 5964

I'd have expected the odisrvreg utility to report "already registered - updating". This leaves a somewhat eery feeling; anyone knowing what is going on, please comment!
I'll update myself on that: the odisrv process does not need to run on both sides - it's supposed to failover. However, I still fail to see how - I even tried kill -9 (all processes), but could not get odisrv to start on the other node.

Let's continue with phase 7: installation of the middle tier:
Machines are fired up, all processes are up-and-running.
Phase 7a: Preliminaries (see phase 5a).
[root@idmhost ~]# groupadd idmown
[root@idmhost ~]# groupadd idminst
[root@idmhost ~]# useradd idmoracle -g idminst -G idmown -c 'Oracle Identity Mgmnt/SSO sw owner'
[root@idmhost ~]# passwd idmoracle
Changing password for user idmoracle.
New UNIX password:
Retype new UNIX password:
passwd: all authentication tokens updated successfully.
[root@idmhost ~]# mkdir /oracle/idm
[root@idmhost ~]# chown idmoracle:idmown /oracle/idm
[root@idmhost ~]# su - idmoracle
[idmoracle@idmhost ~]$ cp /install/Disk1/stage/Response/staticports.ini /oracle/idm/
edit staticports.ini: OID port: 3060, SSL OID port: 3130.

Phase 7b: Install first middle tier (SSO and DAS server).

Now, fire up Cywin X server, and:

frankbo@cs-frank03 ~
$ xhost +
access control disabled, clients can connect from any host

frankbo@cs-frank03 ~
$ ssh idmoracle@idmhost
idmoracle@idmhost's password:
Last login: Sun Jul 8 14:35:34 2007 from dbhost.home.local
[idmoracle@idmhost ~]$ export DISPLAY=192.168.1.104:0.0
[idmoracle@idmhost ~]$ /install/Disk1/runInstaller -invPtrLoc /oracle/idm/oraInventory/oraInst.loc

Fill in the correct settings:


Ditto:


It's still called "Infrastructure", although this is the middle tier:


And I still am not done with the Identity Management Install:


Oh, well, we've been here before...


So let's get started - note I added HA and Replication:


Select the correct file - it needs to pick up the ports actually in use by the OID install (phase 5)


This is an odd one: I am *not* adding a listener, so why this check is executed is beyond me. The resolution is to stop the services on this machine (logon as oidoracle, and issue an opmnctl stopall, or stopproc ias-component=OID)


Once the "error" hurdle is taken, select Cluster:


First install, so I have to create a cluster:


Name it:


Specify correct host; I had the "crossed" setup, so this SSO install (middle tier) will be served by the first install of the Infrastructure, which was on the oidhost:


Specify the password of orcladmin on the OID host:


I make a mistake here - specified the port, as used in metalink note 370458.1. Consequently, I had to change the loadbalancer:

balance -b login.home.local http idm1:7779 % idm2:7779 %


Make up a password, or -better yet- have one generated:


And finally - after a while, and the execution of the (in-)famous root.sh script:


This is what the last screen has to tell:
The following J2EE Applications have been deployed and are accessible at the URLs listed below.

Use the following URL to access the Oracle Enterprise Manager 10g Application Server Control Console :
http://idmhost.home.local:1156

The following information is available in:
/oracle/idm/idm10.1.2/install/setupinfo.txt

Oracle Application Server 10g (10.1.2.0.2) Usernames and Default password information:
Please refer to Oracle Application Server 10g Administrator Guide for more information.

Install Type: Identity Management
Configured Components: Oracle HTTP Server | Oracle Application Server Containers for J2EE | Oracle Application Server Single Sign-On | Oracle Application Server Delegated Administration Service | High Availability and Replication |

A new Oracle Application Server Cluster (Identity Management) has been created named SSOClusterA. The current instance has been joined this cluster at the end of installation.

Load Balancer Servers and ports specified for this instance:
HTTP Load Balancer: login.home.local:
LDAP Load Balancer: oidhost.home.local
SSL Port:3130
Non-SSL Port: 3060

Access URL for Oracle Delegated Administration Services for this instance:
http://login.home.local:80/oiddas

Administrator URL for Oracle Application Server Single-Sign On for this instance:
http://login.home.local:80/pls/orasso

Use the following URL to access the Oracle HTTP Server and the Welcome Page:
http://login.home.local:80
-----------------------------------------
Use the following URL to access the Oracle Enterprise Manager Application Server Control:
http://idmhost.home.local:1156

Instance Name: idm1012_01.idmhost.home.local

Installation of Oracle Application Server Infrastructure is Complete. Please note that any URLs created in this install may not be functional immediately.

Now - let me see if the loadbalancer works.

The defaul (login.home.local) Delegated Administration Service page:


After a successfull login:


After Logout, the node information is shown:


Ok - next step: phase 7c: passwordsI need to synchronize all passwords. One of the installation Wizards did randomize all passwords used in this setup. As connections may float, I do want passwords to be the same on both nodes. The script ssoReplSetup.jar is a Java script, residing in $ORACLE_HOME/sso/lib.
Update:

[oidoracle@oidhost ~]$ cd $ORACLE_HOME/sso/lib
[oidoracle@oidhost lib]$ export LD_LIBRARY_PATH=$ORACLE_HOME/lib32:$LD_LIBRARY_PATH
[oidoracle@oidhost lib]$ echo $LD_LIBRARY_PATH
/oracle/ias/oid10.1.2/lib32:/oracle/ias/oid10.1.2/lib
[oidoracle@oidhost lib]$ $ORACLE_HOME/jdk/bin/java -jar ssoReplSetup.jar -prompt
OracleAS Single Sign-On Replication Setup Tool
Release 10.1.2.0.0
Copyright (c) 2003, 2004 Oracle. All rights reserved.

Reading input paramterers ...

Enter MDS OID hostname : oidhost.home.local

Enter MDS OID port : 3060

Enter MDS OID administrator : cn=orcladmin

Enter MDS OID password : Welcome1

Enter MDS OID SSL Enabled (Y/N) : n

Enter RMS OID hostname : idmhost.home.local

Enter RMS OID port : 3060

Enter RMS OID administrator : cn=orcladmin

Enter RMS OID password : Welcome1

Enter RMS OID SSL Enabled (Y/N) : n

Enter RMS SYS DB password : MANAGER


Done reading parameters.

Contacting OID: ldap://oidhost.home.local:3060 ...
OID context received for MDS admin user, cn=orcladmin

Contacting RMS OID: ldap://idmhost.home.local:3060 ...
OID context received for RMS admin user, cn=orcladmin

MDS DB dn: orclReferenceName=DB1020.CS.NL,cn=IAS Infrastructure Databases,cn=IAS,cn=Products,cn=OracleContext
RMS DB dn: orclReferenceName=OIDREP.CS.NL,cn=IAS Infrastructure Databases,cn=IAS,cn=Products,cn=OracleContext
Starting password synchronization between MDS DB and RMS DB.
ERROR: RMS DB connection failed.
Action: Please check the RMS DB SYS Password.
Exception: java.sql.SQLException: ORA-28009: connection as SYS should be as SYSDBA or SYSOPER

java.sql.SQLException: ORA-28009: connection as SYS should be as SYSDBA or SYSOPER

at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:137)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:304)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:271)
at oracle.jdbc.driver.T4CTTIoauthenticate.receiveOauth(T4CTTIoauthenticate.java:647)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:307)
at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:433)
at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:150)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:31)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:512)
at java.sql.DriverManager.getConnection(DriverManager.java:171)
at oracle.security.sso.server.conf.SyncSSOPwd.syncUpPwds(SyncSSOPwd.java:303)
at oracle.security.sso.server.conf.SyncSSOPwd.main(SyncSSOPwd.java:752)

Checking the password revealed:
SQL> connect sys/manager@db1020 as sysdba
Connected.
SQL> connect sys/manager@oidrep as sysdba
ERROR:
ORA-01017: invalid username/password; logon denied
After changing it, I could logon as sysdba - the error is somewhat unclear - the message is right on spot:
[oidoracle@oidhost lib]$ $ORACLE_HOME/jdk/bin/java -jar ssoReplSetup.jar -prompt
OracleAS Single Sign-On Replication Setup Tool
Release 10.1.2.0.0
Copyright (c) 2003, 2004 Oracle. All rights reserved.

Reading input paramterers ...

Enter MDS OID hostname : oidhost.home.local

Enter MDS OID port : 3060

Enter MDS OID administrator : cn=orcladmin

Enter MDS OID password : Welcome1

Enter MDS OID SSL Enabled (Y/N) : n

Enter RMS OID hostname : idmhost.home.local

Enter RMS OID port : 3060

Enter RMS OID administrator : cn=orcladmin

Enter RMS OID password : Welcome1

Enter RMS OID SSL Enabled (Y/N) : n

Enter RMS SYS DB password : manager


Done reading parameters.

Contacting OID: ldap://oidhost.home.local:3060 ...
OID context received for MDS admin user, cn=orcladmin

Contacting RMS OID: ldap://idmhost.home.local:3060 ...
OID context received for RMS admin user, cn=orcladmin

MDS DB dn: orclReferenceName=DB1020.CS.NL,cn=IAS Infrastructure Databases,cn=IAS,cn=Products,cn=OracleContext
RMS DB dn: orclReferenceName=OIDREP.CS.NL,cn=IAS Infrastructure Databases,cn=IAS,cn=Products,cn=OracleContext
Starting password synchronization between MDS DB and RMS DB.
Creating RMS DB connection ... Done.

Synchronizing the password for orasso ...
MDS - orasso password: *****
Modifying orasso schema pwd value in RMS OID...
Modification of orasso user password in RMS OID successful.
Modifying the orasso user password in secondary database ...
Modification of orasso password in RMS db successful.

Synchronizing the password for orasso_ds ...
MDS - orasso_ds password: *****
Modifying orasso_ds schema pwd value in RMS OID...
Modification of orasso_ds user password in RMS OID successful.
Modifying the orasso_ds user password in secondary database ...
Modification of orasso_ds password in RMS db successful.

Synchronizing the password for orasso_pa ...
MDS - orasso_pa password: *****
Modifying orasso_pa schema pwd value in RMS OID...
Modification of orasso_pa user password in RMS OID successful.
Modifying the orasso_pa user password in secondary database ...
Modification of orasso_pa password in RMS db successful.

Synchronizing the password for orasso_public ...
MDS - orasso_public password: *****
Modifying orasso_public schema pwd value in RMS OID...
Modification of orasso_public user password in RMS OID successful.
Modifying the orasso_public user password in secondary database ...
Modification of orasso_public password in RMS db successful.

Synchronizing the password for orasso_ps ...
MDS - orasso_ps password: *****
Modifying orasso_ps schema pwd value in RMS OID...
Modification of orasso_ps user password in RMS OID successful.
Modifying the orasso_ps user password in secondary database ...
Modification of orasso_ps password in RMS db successful.

Setting SSO server preferences table in RMS DB ...
Connected to MDS DB as ORASSO user.
Retrieved SSO_SERVER pwd: *****
Decrypted SSO_SERVER pwd: *****
Connected to RMS DB as ORASSO user.

Setting SSO server preferences table in RMS DB ...
Connected to MDS DB as ORASSO user.
MDS node LDAP connection SSL usage: Y
ERROR: MDS node is configured to use LDAP over SSL.
ACTION: Please provide LDAP SSL port for the RMS node.

The last line indicates I should use the SSL port (3130):

[oidoracle@oidhost lib]$ $ORACLE_HOME/jdk/bin/java -jar ssoReplSetup.jar -prompt
OracleAS Single Sign-On Replication Setup Tool
Release 10.1.2.0.0
Copyright (c) 2003, 2004 Oracle. All rights reserved.

Reading input paramterers ...

Enter MDS OID hostname : oidhost.home.local

Enter MDS OID port : 3130

Enter MDS OID administrator : cn=orcladmin

Enter MDS OID password : Welcome1

Enter MDS OID SSL Enabled (Y/N) : Y

Enter RMS OID hostname : idmhost.home.local

Enter RMS OID port : 3130

Enter RMS OID administrator : cn=orcladmin

Enter RMS OID password : Welcome1

Enter RMS OID SSL Enabled (Y/N) : Y

Enter RMS SYS DB password : manager


Done reading parameters.

Contacting OID: ldap://oidhost.home.local:3130 ...
OID context received for MDS admin user, cn=orcladmin

Contacting RMS OID: ldap://idmhost.home.local:3130 ...
OID context received for RMS admin user, cn=orcladmin

MDS DB dn: orclReferenceName=DB1020.CS.NL,cn=IAS Infrastructure Databases,cn=IAS,cn=Products,cn=OracleContext
RMS DB dn: orclReferenceName=OIDREP.CS.NL,cn=IAS Infrastructure Databases,cn=IAS,cn=Products,cn=OracleContext
Starting password synchronization between MDS DB and RMS DB.
Creating RMS DB connection ... Done.

Synchronizing the password for orasso ...
MDS - orasso password: *****
Modifying orasso schema pwd value in RMS OID...
Modification of orasso user password in RMS OID successful.
Modifying the orasso user password in secondary database ...
Modification of orasso password in RMS db successful.

Synchronizing the password for orasso_ds ...
MDS - orasso_ds password: *****
Modifying orasso_ds schema pwd value in RMS OID...
Modification of orasso_ds user password in RMS OID successful.
Modifying the orasso_ds user password in secondary database ...
Modification of orasso_ds password in RMS db successful.

Synchronizing the password for orasso_pa ...
MDS - orasso_pa password: *****
Modifying orasso_pa schema pwd value in RMS OID...
Modification of orasso_pa user password in RMS OID successful.
Modifying the orasso_pa user password in secondary database ...
Modification of orasso_pa password in RMS db successful.

Synchronizing the password for orasso_public ...
MDS - orasso_public password: *****
Modifying orasso_public schema pwd value in RMS OID...
Modification of orasso_public user password in RMS OID successful.
Modifying the orasso_public user password in secondary database ...
Modification of orasso_public password in RMS db successful.

Synchronizing the password for orasso_ps ...
MDS - orasso_ps password: *****
Modifying orasso_ps schema pwd value in RMS OID...
Modification of orasso_ps user password in RMS OID successful.
Modifying the orasso_ps user password in secondary database ...
Modification of orasso_ps password in RMS db successful.

Setting SSO server preferences table in RMS DB ...
Connected to MDS DB as ORASSO user.
Retrieved SSO_SERVER pwd: *****
Decrypted SSO_SERVER pwd: *****
Connected to RMS DB as ORASSO user.

Setting SSO server preferences table in RMS DB ...
Connected to MDS DB as ORASSO user.
MDS node LDAP connection SSL usage: Y
Setting OID configurations in RMS DB Done.

Setting up the SSO Server site token in the prefs table...
Updating SSO preference store for the SSO Server site token...
SSO Replication configuration successfully finished.


Same thing needs to be done on the replicated site, idmhost.home.local. I found it not very clear whether this should be done in the middle tier, or in the infrastructure - the notes suggest the first, and so does the point in time: after the first middle-tier install.
Rest assured: it should run from the infrastructure - the sites, where the OID processes and replication run.
All that is left to install now, is the last middle tier:
[root@oidhost ~]# mkdir -p /oracle/idm/oraInventory
[root@oidhost ~]# cd /oracle
[root@oidhost oracle]# chown -R idmoracle:idminst idm


The following J2EE Applications have been deployed and are accessible at the URLs listed below.

Use the following URL to access the Oracle Enterprise Manager 10g Application Server Control Console :
http://oidhost.home.local:1156

The following information is available in:
/oracle/idm/idm10.1.2/install/setupinfo.txt

Oracle Application Server 10g (10.1.2.0.2) Usernames and Default password information:

Please refer to Oracle Application Server 10g Administrator Guide for more information.

Install Type: Identity Management

Configured Components: Oracle HTTP Server | Oracle Application Server Containers for J2EE | Oracle Application Server Single Sign-On | Oracle Application Server Delegated Administration Service | High Availability and Replication |

A new Oracle Application Server Cluster (Identity Management) has been created named SSOClusterB. The current instance has been joined this cluster at the end of installation.

Load Balancer Servers and ports specified for this instance:
HTTP Load Balancer: login.home.local:
LDAP Load Balancer: idmhost.home.local
SSL Port:3130
Non-SSL Port: 3060

Access URL for Oracle Delegated Administration Services for this instance:
http://login.home.local:80/oiddas

Administrator URL for Oracle Application Server Single-Sign On for this instance:
http://login.home.local:80/pls/orasso

Use the following URL to access the Oracle HTTP Server and the Welcome Page:
http://login.home.local:80
-----------------------------------------
Use the following URL to access the Oracle Enterprise Manager Application Server Control:
http://oidhost.home.local:1156

Instance Name: idm1012_02.oidhost.home.local

Installation of Oracle Application Server Infrastructure is Complete. Please note that any URLs created in this install may not be functional immediately.


The installation is the same as the first one, except for some names, that are different (obviously): the cluster is called SSOClusterB (could have been the same, by the way), the ldapserver is idmhost.home.local (I am installing on oidhost!), so I will not post any screendumps of that.
Instead, stay tuned for replication woes, and usage notes.

Last and Final Update:
To show that the whole things is two-fold:


There you have it - two partner applications.

In a nutshell:
  1. Install and patch the database software tree(s).
  2. Create a database, altering default the settings to ones, fit for a Repository. If not done now, the Metadat Creation Repository Assistant (MRCA) will force you.
  3. Run the MRCA against the newly created database.
  4. Clone to create the replication database (or reuse the scripts and rerun MRCA)
  5. Install the first Infrastructure. Options: OID and DIP. Use main database as repository database.
  6. Install the second Infrastructure. Options: OID, DIP and HA/Replication, use first infrastructure OID setup as reference. Use replica database for repository database.
  7. Configure your network:
    1. make sure you can start SQL*Plus from both database, and both Infrastructure environments. Als make sure, you can use shorthand, as well as the fully qualified tns-alias. This step is crucial!
    2. Also, make sure you have your loadbalacer and naming (DNS or other) in order.
  8. Setup the OID replication, using the remtool ($ORACLE_HOME/bin/remtool -asrsetup -v)
  9. Stop and start (using $ORACLE_HOME/opmn/bin/opmnctl) all processes on both Infrastructure installations.
  10. Start the replication processes; first time only by hand, using oidctl, on both Infrastructure installations.
  11. Check replication by adding on an entry in one OID environment, wait until it appears in the other. then, delete from the other, and check whther it disappears from the first.
  12. Install the first Middle Tier (Single Sign On/Delegated Administration Services). Oddly enough, it is still an infrastructure install. Select SSO, DAS and HA, create a new cluster. Specify the first OID install for LDAP, and your loadbalancer.
  13. Synchronize passwords, generated at random during the installation, across both infrastructures. Use ssoReplSetup.jar -prompt on both Infrastructure installs. Mind the LD_LIBRARY_PATH.
  14. Install the second Middle Tier (SSO/DAS). Similar to first install.
Can be done in two days, provided everything is prepared, root access available and no platform specific bugs are ran into. And enough coffee...