mount Oracle

Just curious:

mountd 20048/tcp NFS mount protocol
mountd 20048/udp NFS mount protocol
# Nicolas Williams <Nicolas.Williams_at_oracle.com> 09 August 2010

ORA-12638: Status 82 "Not a bug"

One thing that completely spoils the previous entry on Kerberos authentication in the database, is the state of denial Oracle is in. It seems impossible to admit there's a bug in SQL*Net in the sense there's no fallback for username/password authentication when Kerberos fails.

ORA-12638 [Credential retrieval failed]

What's the case? Well, after you have set up Kerberos authentication, your database links seize working with ORA-12638: Credential retrieval failed, as soon as both databases are Kerberos aware.

Now, of course this is a situation you do not want. You want administrators and users to log in using their working environment credentials, but you rely on named accounts, and passwords, for database links.
Alas. Here comes Bug 6341668: CLIENT CAN NOT CONNECT TO DATABASE BY PASSWORD IF ANY KERBEROS PROBLEM OCCURS. Current status: 82 "Closed, not a bug". Al least someone else tried Kerberos authentication...
This is in contradiction with another Kerberos-related bug, # 2647883: Kerberos current user database links do not work (See Doc ID 2647883.8), where the actual workaround is ... use fixed user database links (!).
Catch-22, eh?

I hate it when I get confronted with this rubbish on Friday afternoons. It ruins my weekend!
Oh - might anyone wonder: this is cross platform, 10.2.0.4.

One other reference on Oracle Support: TNS-12638 when database authentication tried on Kerberos setup (Doc ID 972896.1)

Updates

ASO is standard

There's a note stating the Advanced Security Option is now standard with the Enterprise Database Install. Makes you wonder: is it included in the standard license fee, then?

Patch 10.2.0.5

Took the time to update the 10.2.0.4 environment I tested the scenario in to 10.2.0.5, which does have some Kerberos related bugs resolved. The result of all this is not only a ruined Sunday, it's also not resolving anything.

C:\>sqlplus /@db10
SQL*Plus: Release 10.2.0.5.0 - Production on Sun Sep 26 18:32:25 2010
Copyright (c) 1982, 2010, Oracle. All Rights Reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - Production
With the Partitioning, Data Mining and Real Application Testing options

SQL> select sysdate from dual@fvb;
select sysdate from dual@fvb
*
ERROR at line 1:
ORA-12638: Credential retrieval failed

Only after removing the services from sqlnet.ora on the server side, it works:

[ora10@db1 admin]$ cat sqlnet.ora
names.default_domain=home.local
# sqlnet.authentication_services=(beq, kerberos5)
sqlnet.kerberos5_conf=/oracle/db10/admin/krb5.conf
sqlnet.kerberos5_keytab=/oracle/db10/admin/db1.keytab
sqlnet.authentication_kerberos5_service=orcl
sqlnet.kerberos5_conf_mit=true

Which makes the same query on the client work... instantaneous!

SQL> select sysdate from dual@fvb;

SYSDATE
---------
26-SEP-10

It's getting weirder

I stumbled upon this, trying to find a workable solution on maintainting two TNS_ADMIN locations on the server. A similar problem has been logged as Bug 5054469: "DATABASE AUTHENTICATED USERS REQUIRE KERBEROS CREDENTIAL CACHE TO CONNECT", which has been closed as duplicate of (internal) bug 9056628.
Explain: how can a bug# 5 million-and-something be closed as a duplicate of bug# almost-twice-as-high? Makes me wonder.
And the best is this: Doc ID 972896.1 (TNS-12638 when database authentication tried on Kerberos setup) states "This has been an issue in the past that the authentication services are not skipped to next when first fails".
Yeah right: has been... in the past...

Synchronize OID 10.1.4.3 with Active Directory

There's quite a difference between 10.1.4.1 (and earlier versions) and 10.1.4.3 in the syncing business. You may now use multiple profiles quite easily and even have separate jobs for each profile. This would allow for a multi-AD-domain company to allow for different frequencies of synchronization: have one, well-know busy AD Domain synchronize very frequently, and other less frequent.
You may also enable or disable profiles by simply removing them from the running sync job(s); that process is called disassociation.

Receipe

So, what do I need?
1 Active Directory Installation
1 Oracle Identity Management Installation

I go both, one running VMWare (MS AD - I had it shipped as VMWare image), one Oracle, installed under VirtualBox.

Start it up

As the Oracle installation is as basic as it can be, it needs starting up:

frank@frank-cs03:~$ ssh oracle@192.168.1.222
oracle@192.168.1.222's password:
Last login: Fr Sep 10 16:43:24 2010
[oracle@oracleas ~]$ . oraenv
ORACLE_SID = [oracle] ? iasdb
[oracle@oracleas ~]$ lsnrctl start

LSNRCTL for Linux: Version 10.1.0.5.0 - Production on 29-DEC-2010 09:46:40

Copyright (c) 1991, 2004, Oracle. All rights reserved.

Starting /oracle/ias/10.1.4/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 10.1.0.5.0 - Production
System parameter file is /oracle/ias/10.1.4/network/admin/listener.ora
Log messages written to /oracle/ias/10.1.4/network/log/listener.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=oracleas)(PORT=1521)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracleas)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 10.1.0.5.0 - Production
Start Date 29-DEC-2010 09:46:41
Uptime 0 days 0 hr. 0 min. 0 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /oracle/ias/10.1.4/network/admin/listener.ora
Listener Log File /oracle/ias/10.1.4/network/log/listener.log
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=oracleas)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

I need a tcp based listener, because of the OID/ldap stack, that uses tcp. Otherwise, I would not have gone through all of the tcp stack, but use Direct Calls, or even better: bequeth, which does not need a listener at all. Anyway - open the database:

[oracle@oracleas ~]$ sqlplus / as sysdba

SQL*Plus: Release 10.1.0.5.0 - Production on Wed Dec 29 09:47:33 2010

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 281018368 bytes
Fixed Size 779000 bytes
Variable Size 229645576 bytes
Database Buffers 50331648 bytes
Redo Buffers 262144 bytes
Database mounted.
Database opened.
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.5.0 - Production
With the Partitioning, OLAP and Data Mining options

After all that, it is time to get ldap running. It needs to connect to the database, so this is the correct starting order:

[oracle@oracleas ~]$ $ORACLE_HOME/opmn/bin/opmnctl startall
opmnctl: starting opmn and all managed processes...

See if all is started:

[oracle@oracleas ~]$ $ORACLE_HOME/opmn/bin/opmnctl status

Processes in Instance: ias_1014.oracleas
-------------------+--------------------+---------+---------
ias-component | process-type | pid | status
-------------------+--------------------+---------+---------
DSA | DSA | N/A | Down
LogLoader | logloaderd | N/A | Down
dcm-daemon | dcm-daemon | N/A | Down
OC4J | OC4J_SECURITY | 2811 | Alive
HTTP_Server | HTTP_Server | 2726 | Alive
OID | OID | 2731 | Alive

If you would like to know which ports are used by the http server, use the "-l" option of the opmnctl command (-l stands for "long").
Now, with the OID, there's another check you ought to be aware of. Similar to the way the Oracle Process Manager (opmn) monitors, OID has it's own set of monitors. Activity of these can be checked by using the ldapcheck command:

[oracle@oracleas ~]$ $ORACLE_HOME/ldap/bin/ldapcheck

Checking Oracle Internet Directory Processes ...ALL

Process oidmon is Alive as PID 2731
Process oidldapd is Alive as PID 2749
Process oidldapd is Alive as PID 2783
Not Running ---- Process oidrepld
Process odisrv is Alive as PID 2770

oidmon (PID 2731) is the OID Monitoring process itself; it is the process, that is actually monitored by opmn. The two oidldapd processes are two LDAP Daemons. Other than that, there's an inactive Replication process, oidrepld, that is only used in Enterprise-class setups, as I described here.
The final process is the one, I'm interested in: it's the Oracle Directory Integration service, odisrv for short.

What if I have no ldapcheck?

Well, I did not, either. You can download diptester from here, unzip it, and rename "ldapcheck_for_HPUX" to ldapcheck, do a chmod 740, and you're in business.

What id ldapcheck returns

ps: illegal option -- o

Apply same medicin as "What if I do not have ldapcheck"

Synchronize!

Now, in order to get OID to synchronize from AD, I need a few things:

• a highly privileged account on Active Directory
• Names, or IP-addresses, of servers involved
• an understanding of mapping, with base of OID as well as AD

As for the account, I'll use my own account, which is member of the administrators group - do not use this in production! But you do need an account, that is allowed to query the "Deleted users" section - and that requires quite some privileges. If you cannot see "Deleted Users", users can be removed from AD, but will always remain in OID. And that is a security issue...

IP addresses: The oracle iAS runs is called oracleas.home.local at 192.168.1.222, the PDC is located at 192.168.1.233.

Mapping: well, whoever invented that should be prosecuted. As long as you keep it basic, and simple, there are few problems. As soon as you company is operating multiple domains, it is a RPITA to get things running smoothly.

Profiles

Mapping is based on profiles. The fastest way to get things running is to take a copy of $ORACLE_HOME/ldap/odi/conf/activechg.map.master and use that as basis.
The base in OID is dc=bortel,dc=home, as you can see from this picture of the OID Aministration tool, oidadmin:

This is the correct entry, as this entry holds the users. Same is true for Active Directory, although I know no other way than to query the darn thing:

C:\Documents and Settings\frank>ldapsearch -Z -h w2k-pdc -p 3268 cn=frank cn
CN=frank,CN=Users,DC=home,DC=local
cn=frank

What I am doing here, is query my own credentials, using the built-in security (-Z), the localhost as ldap host (-h w2k-pdc), and query the Global Catalog port (-p 3268).
Anyway - my AD mapping is dc=home, dc=local.

Mapping rules

In the mapping file, this results in the first line being:

cn=users,dc=home,dc=local:cn=users,dc=bortel,dc=home

In words: maps everything, found under cn=users,dc=home,dc=local (in Active Directory, as it is left of the colon) to cn=users,dc=bortel,dc=home (in Oracle Internet Directory, as it is right of the colon).

These are the domain rules.

If you have more domains, just add mapping rules. If you have organized your domain in Units (Organizational Units, or "ou" in LDAP-speak), start mapping rules at that level, and add, or omit to your needs.

dipadmin

Now, a new tool needs to be used, the dispassistant. User is dipadmin - password of dipadmin is identical to the password of orcladmin:


New in this version is the use of Connector Profiles, and profile Sets.
Due to badly documented features, and some documentation bugs it is best to create a new Connector Group.
First of all, rename the Connector Groups "configset1" to "standardgroup". The original name will just create misunderstanding, as the are configsets in use with OID.


Then, create the group you will be using for your actions - the "standardgroup" will remain as the group holding all defaults, but will not actually be used. My new group is called "demo":


Next, go to the standardgroup, and disassociate ActiveChgImp:


Then, select the newly created group "demo", and associate the profile:


Note, the profile is still Disabled. Edit the profile, and fill in the connection defaults on the first tab:


On the third tab, make sure the filter properties do not have quotes around them - it's a bug:


On the last tab, I always change "Continue on Error" to Yes; I have no tools to monitor whether every change gets into OID correctly. Most of the errors I see, are on duplicated entries anyway.
You may also want to change the frequency of synchronization: every 60 seconds seems on the high side, escpecially if you have 7 profiles, like my customer, one of which connects to a Active Directory with 13,000 users.


Modify the profile mapping file to your liking, and load it into the profile:

[oracle@oracleas ~]$ dipassistant mp -D cn=orcladmin -profile=activechgimp odip.profile.mapfile=demo_user.map
Password:
Profile successfully modified.
[oracle@oracleas ~]$

Bootstrap

The first, initial (bulk) load of the accounts is call bootstrapping:

[oracle@oracleas ~]$ dipassistant mp -D cn=orcladmin -profile=activechgimp odip.profile.mapfile=demo_user.map
Password:
Profile successfully modified.
[oracle@oracleas ~]$

Results of this process may be found in $ORACLE_HOME/ldap/odi/log/bootstrap.log and ~.trc

You can also inspect the OID with the oidadmin tool:


(and see there's a problem with krbPrincipalName...)

Activate

Now, all that's left is to start the process of syncs every x seconds. In order to do that, I'll need to start the connectorgroup:

[oracle@oracleas ~]$ $ORACLE_HOME/ldap/bin/ldapcheck

Checking Oracle Internet Directory Processes ...ALL

Process oidmon is Alive as PID 2513
Process oidldapd is Alive as PID 2519
Process oidldapd is Alive as PID 2526
Not Running ---- Process oidrepld
Process odisrv is Alive as PID 2521
[oracle@oracleas ~]$ oidctl connect=iasdb server=odisrv instance=2 configset=1 flags="grpid=demo" start
NLS_LANG not set in environment
Setting NLS_LANG to AMERICAN_AMERICA.AL32UTF8
oidctl:Waiting for oidmon to start ODISRV (instance=2)
oidctl:Waiting for oidmon to start ODISRV (instance=2)
oidctl:Started ODISRV (instance=2) with PID : 21496 successfully

[oracle@oracleas ~]$ $ORACLE_HOME/ldap/bin/ldapcheck

Checking Oracle Internet Directory Processes ...ALL

Process oidmon is Alive as PID 2513
Process oidldapd is Alive as PID 2519
Process oidldapd is Alive as PID 2526
Not Running ---- Process oidrepld
Process odisrv is Alive as PID 2521
Process odisrv is Alive as PID 21496
[oracle@oracleas ~]$

There are two odi processes now. The second one is the AD-OID synchronisation.
Logging of this process is in the same directory as the bootstrap: $ORACLE_HOME/ldap/odi/log.
You will see the timestamp changing (and the file will grow.

Now, after adding a new user in AD, I can quey these:

[oracle@oracleas ~]$ ldapsearch cn=frank* uid
cn=frank,cn=users,dc=bortel,dc=home

cn=frank w.j.. van bortel,cn=users,dc=bortel,dc=home
uid=frank2@home.local

And that proves, the synchronisation is working!

Groups, and all that

This is to be added: how to sync groups, and split these. I will also mention some considerations about syncing, especially on multiple domains.

Wall-to-wall SSO and EUS

After introducing Kerberos, and getting SSO to the database working, it's time for some excitement: combine Single Sign On (SSO) with Enterprise User Security (EUS).
Sure, I blogged about Enterprise Security a long time ago (See the "Enterprise Security" entries of 2005), but there I used certificates; in fact another identity.
In this entry, some of the 2005 stuff returns, but parts will differ.

The latest and the greatest.

For those of you how wonder why I bother to blog about this "ancient" version, read the statement of direction on SSO server (note 979951.1 on MOS). Basically, when you use Portal, Forms or Discoverer, you need OSSO 10GR3 (even Forms 11G).
It's not that ancient when it will be around for some time...

Goal.

What I want, is the same a the previous entry on SSO, but I do not wish to create users: I want to use EUS instead.

Preparations.

In addition to what's already in place, I need a complete SSO infrastructure.
The brave may want to do this Enterprise-sized setup, for now I'll just go with the "all-in-one" Identity Management install. Even patched it to 10.1.4.3.

Steps

In order to achieve all this, I would need to:

1. Set up synchronization between Active Directory and OID
2. Set up Enterprise Security in the database(s)
   
3. Set up a trusted relation between OSSO Application Server and AD
4. As a bonus: try to persuade Forms menu security to use EUS

I will blog about these - stay tuned...

Single Sign On to the Database (or WNA for SQL*Plus)

In this entry, I'll demonstrate how easy it is to do Kerberos authentication to the database, or -rephrasing that- how to do WNA in the database.
The goal of this exercise will be to allow users to log on to a remote database using the following syntax:

sqlplus /@db10g

Scenario

I like to think major sites have serious machines and ditto operating systems. Workstations are uncannily often based on MS Windows, so I'll use MS Windows for a client, and Linux for database server.

Prerequisites

Both client and server need to have the Advanced Security Option installed, for which Oracle wants to get paid. Just read the license conditions: if you want to do Kerberos, you need to pay.

Checks

Now, how do you check whether you have that installed? On *ixes, you can use the command adapters:

Installed Oracle Advanced Security options are:

RC4 40-bit encryption
RC4 56-bit encryption
RC4 128-bit encryption
RC4 256-bit encryption
DES40 40-bit encryption
DES 56-bit encryption
3DES 112-bit encryption
3DES 168-bit encryption
AES 128-bit encryption
AES 192-bit encryption
AES 256-bit encryption
MD5 crypto-checksumming
SHA-1 crypto-checksumming
Kerberos v5 authentication
RADIUS authentication
[ora10@db1 ~]$

According to the documentation, that is proof. I have found that it is not always the case, and you may find yourself without the Oracle Kerberos utilities. Better check if these are available, they sould be:

[ora10@db1 ~]$ ls $ORACLE_HOME/bin/ok*
/oracle/db10/10GRel2/bin/okdstry /oracle/db10/10GRel2/bin/okinit
/oracle/db10/10GRel2/bin/oklist /oracle/db10/10GRel2/bin/okdstry0
/oracle/db10/10GRel2/bin/okinit0 /oracle/db10/10GRel2/bin/oklist0

That concludes checks on the software stack on the server. If you miss anything: install the Advanced Security Option!

Client checks

For MS Windows clients, there is unfortunately not much else to do than to fire up OUI, and list the installed options. Fore mentioned Kerberos utilities should also be available on the client, so you may want to check that out as well.

Hands on

Service Principal

First thing to do, is to allow the database server to get Kerberos tickets. In order to do that, I need a Kerberos service principal. In order to do that, I need to create a domain user on my MS Windows 2000 Domain Controller:

I have chosen the name of the instance, but that does not really matter. You may want to use a more generic user name if you have an environment with many instances per machine.
Make sure you set "Use DES encryption" and "Do not require preauthentication", as shown here:

You may have to patch your MicroSoft Active Directory to support this. Check out Knowledge Base article 833708 on how to do this in the registry, or apply SP 1.
I think everybody has SP1 (or better) - I did not have to change anything on my PDC image, but this may cause problems.

Keytab file

Now that I created a user, I must transfer the credentials to the database server. That involves exporting keys to a file, and it is done using this statement on the PDC (or member domain controller):

ktpass -princ service/username@AD.DOMAIN -mapuser username -pass APassWord -DesOnly -crypto des-cbc-md5 -ptype KRB5_NT_PRINCIPAL -out c:\serv.keytab

This is an actual session:


The beauty is, that you may forget the password: you will never need it again. In fact, you need not know the password, unless things don't work, and you want to troubleshoot.
You should remember the service name: you need to specify that in configuration files. I use "orcl" in this example - you may want to use a more generic name, like krb5srv.
Oracle MOS Document states you can only use -crypto des-cbc-crc for encrytion; this is incomplete; you can also use -crypto des-cbc-md5, as I did. (you need md5 for HTTP principals, but that's an other story).

Prepare server

Binary transfer the keytab file to the server. These files are called keytab files, as they contain a table of keys - the keytab for short.
Logon to the database server an cd to your TNS_ADMIN directory. Which happens to be $ORACLE_HOME/network/admin, if you do not use $TNS_ADMIN.
Alter the sqlnet.ora file to look like:

[ora10@db1 admin]$ cat sqlnet.ora
sqlnet.authentication_services=(beq, kerberos5)
sqlnet.kerberos5_conf=/oracle/db10/admin/krb5.conf
sqlnet.kerberos5_keytab=/oracle/db10/admin/db1.keytab
sqlnet.authentication_kerberos5_service=orcl
sqlnet.kerberos5_conf_mit=true

Make sure the variables point to the correct locations of the keytab file and the (general) Kerberos configuration file. This configuration file contains:

[ora10@db1 admin]$ cat /oracle/db10/admin/krb5.conf
[libdefaults]
default_realm = HOME.LOCAL
[realms]
HOME.LOCAL = {
kdc=192.168.1.250
}
[domain_realm]
.home.local = HOME.LOCAL
home.local = HOME.LOCAL

Mind you, I am completely bypassing the fact, your server should be configured to support Kerberos, i.e. without support for Kerberos on OS level, this is less likely to succeed.

Prepare Client

Client side is about the same as the server, apart from the service principal/keytab file. Change sqlnet.ora:

SQLNET.AUTHENTICATION_SERVICES= (all)
NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)
sqlnet.kerberos5_conf=C:\oracle\clt10g\network\admin\krb5.conf
sqlnet.kerberos5_conf_mit=true
sqlnet.authentication_kerberos5_service=orcl
# sqlnet.kerberos5_cc_name=C:\oracle\clt10g\network\admin\krbcache
sqlnet.kerberos5_cc_name=OSMSFT://

Note the last line: it is MS specific, and allows the use of the MS Windows internal Kerberos cache, instead of a file based one (as specified the line above).
Note however, using the internal cache will disallow the use of Oracle Kerberos utilities; these will err off with "OSD error"

The Kerberos configuration file is similar to the one on the server, and connects realms to domains:

[libdefaults]
default_realm = HOME.LOCAL
[realms]
HOME.LOCAL = {
kdc = w2k-pdc.home.local
}
[domain_realm]
.home.local = HOME.LOCAL
home.local = HOME.LOCAL

You may now test the Kerberos setup. You should be able to get a ticket from the Kerberos Ticket Granting Server (which is MS Active Directory, in this case).
Try to get a ticket for a known MS domain account, I can log on to the domain "home.local" as frank, that would make my account "frank@HOME.LOCAL" - and be careful, Kerberos is strangely case sensitive! The oklist utility reveals:

Ticket cache: win2kcc
Default principal: frank@HOME.LOCAL

Valid Starting Expires Principal
08-Sep-2010 12:56:03 08-Sep-2010 22:56:03 krbtgt/HOME.LOCAL@HOME.LOCAL renew until 15-Sep-2010 12:56:03
08-Sep-2010 12:56:03 08-Sep-2010 22:56:03 krbtgt/HOME.LOCAL@HOME.LOCAL renew until 15-Sep-2010 12:56:03
08-Sep-2010 13:28:28 08-Sep-2010 22:56:03 orcl/db1.home.local@HOME.LOCAL renew until 15-Sep-2010 12:56:03
08-Sep-2010 12:56:04 08-Sep-2010 22:56:03 W2K-PDC$@HOME.LOCAL renew until 15-Sep-2010 12:56:03
08-Sep-2010 12:56:04 08-Sep-2010 22:56:03 ldap/w2k-pdc.home.local/home.local@HOME.LOCAL renew until 15-Sep-2010 12:56:03

Now, create a database user "FRANK@HOME.LOCAL", in uppercase, and quoted, as Oracle uses the commercial at-sign for special purposes. You should now be able to login:

C:\Documents and Settings\frank>sqlplus /@db10

SQL*Plus: Release 10.2.0.5.0 - Production on Wed Sep 8 13:51:56 2010
Copyright (c) 1982, 2010, Oracle. All Rights Reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, Data Mining and Real Application Testing options

SQL> show user
USER is "FRANK@HOME.LOCAL"
SQL>

Stay tuned

Of course, things start getting interesting when adding OID and EUS. Wall-to-wall SSO!
Later...

Over the top

As much as I like privacy (google changed it's privacy statement just now, btw), I hate any form of censorship on the internet.
RIM seems to give up on privacy, by allowing governments to decrypt Blackberry traffic.
Google is not just content with our search and surf behaviour on the internet, it now wants to know what we're talking about and introduces free phone services.
And Apple is censoring ping.

Sigh. What a world. Next entry will be on Oracle, and SSO to the database.
Update: Craigslist does censor, too.

[Edit]Seems DRM is broken... Someone published a master key...[/edit]