Asrock E3C224DI-14S IPMI not reachable

Storage Server: Documentation missing

There's definately some documentation missing on the IPMI settings. I managed to lock myself out of the IPMI (also know as 'Integrated Light Out', or ILO) management interface. Not sure how I managed to do that, but in the quest to find out how to restore devine powers, I noticed quite a lot of people suffered from this. And, the solution is quite simple, when you know it. As usual...

Two configurations

The cause of error probably was me updating the network configuration, using the dashboard, instead of using the BIOS update.

IPMI configuration at BIOS
Please note, in the BIOS, you have eth0 and eth1. Eth0 usually is the first interface, so when you assume this would be the IPMI interface, you assumed as I.

Link to Network configuration on the dashboard

Which is completely and utterly wrong... Eth1, which has the label IPMI for a reason, is the correct one, and is found as Port 8 (above the 2 USB ports).

what's with the eth0/eth1, then?

It turns out there's a nifty, quite undocumented feature, of IPMI fallback. the BIOS eth0/NCSI item can be used as IPMI fallback - in case eth1 is not connected... I found this because after the lock out, I could actually use the IPMI when all cables were plugged in. The NCSI port is port 6, or LAN1 (designated as such in the manual), or eth0 (as seen in the BIOS BMC configuration). For completeness sake, LAN2 is not mentioned in the BIOS, only in the Megarac SP configuration (and hard to find).

My recommendation

Stay away from the megarac SP network configration items. Use the BIOS, which takes precedence over the Mearace SP settings anyway, only, and only configure IPMI/eth1 for a fixed ip-adddress. You can always use arp -a to find out the DCHP-assigned ip address to the other interfaces. You can find the MAC-addresses of your LAN1 and LAN2 interfaces at the BMC configuration section of the BIOS, under BMC MAC Restore Tool:

Hope this helps anyone.

Storage server

Storage server

Hardware

Aiming at 2 VDEVs of 5 or 6 disks each, I'd need a motherboard capable of running 12 disks.I used a SuperMicro board in the ESXi build, mainly because virtualization using bare metal hypervisors was quite new to me. However, these boards have quite a steep price.

There's a new motherboard by SuperMicro, that screams NAS, but that has not yet hit the shops.

So, I ended up with:

• 2.5" Toshiba 3TB harddisks (MQ03ABB300)
• 3.5" Toshiba 3TB harddisks
• Asrock E3C224D4i-14S motherboard, supplied with 14 (!) SATA ports
• 32GB 1600MHz ECC ram (Kingston KVR16E11K4/32)
• Define R5 by Fractal Design
• Pentium G3258 by Intel
• G-360 Seasonic power supply
• EVO850 by Samsung

Assembly

It starts off with placing the processor and memory on the motherboard. This is best done outside the case:

One of the reasons I love Fractal cases is the disks cages; not only can you replace the cage with hot plug ones, you may also relocate one or both cages. And the disk frames just slide out - no tools needed.

And, there's room for two SSD's at the back of the motherboard:

The case assemply starts with adding the power supply.

Then, place the motherboard, and attach the power cables. Some cable management is in order, but will be done after all disks have been installed and hooked up.

Time to put these components to the test. Download Memtestx86, or the commercial version, and let it run for a while.

OK, that'll do pig, that'll do.

To be continued with part two of the storage server: Firmware

Things to do after you cloned a Virtual Machine

Clean up a cloned VM

After you made a clone of your (base) VM, you will need to do some stuff.

MAC-address

First of all, I suspect you have a different MAC-address than the original machine. VMWare does that, as long as you have your MAC address assigned automatically. VirtualBox will ask you whether to re-initialize the MAC-address while cloning.
The problem is the udev process, responsable for handling devices. This uses confuguration files, located in/etc/udev/rules.d directory. The file
70-persistent-net.rules will have an entry, based on your original machine. An entry looks like:

# PCI device 0x15ad:0x07b0 (vmxnet3) (custom name provided by external tool) SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:71:3c:be", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

You see the MAC-address, as wel as the assigned link (eth0). I already altered this line to reflect the correct MAC-address, and link name. The easiest way turns out to be to remove this file completely; it will be re-generated when absent:

cd /etc/udev/rules.d/ rm -f 70-persistent-net.rules reboot now reboot

eth0 or eth1

Another problem may be the fact your assigned link names went wrong; you may have extra rules in your udev file for eth1. And, your existing eth0 may still carry the wrong MAC-address, which may cause errors like

ifup eth0 eth0 does not seem to be present,delaying initialization.

Check if your MAC-addresses are still lingering around with:

/sbin/ifconfig | grep "^eth" eth0 Link encap:Ethernet HWaddr 00:0C:29:71:3C:BE

This is a correct output - MAC-addresses match. If not, change the /etc/sysconfig/network-scripts/ifcfg-eth0 file to reflect the correct MAC-address:

DEVICE=eth0 TYPE=Ethernet ONBOOT=yes NM_CONTROLLED=yes BOOTPROTO=none HWADDR=00:0C:29:71:3C:BE

Depending on the number of NIC's, ifcfg-eth1 may require some tweaking, as well. You should now successfully be able to start networking services. 

hostname

Change the hostname; edit /etc/sysconfig/network, and adapt /etc/hosts

Now, here's an idea...

Gaining control

Or rather - regaining control. Over my own data, and what's done with it.

Currently, I use several services, of which I know they are monitored. Several of these services fall under US legislation, although I'm not a US citizen. This allows several agencies to go through my documents, email and other stuff, whether I like that or not (I do not).

Of course, for some of this, I gave permission - blogging on a google platform undoubly allows google to scan, "in order to enhance services rendered". Or something similar. Using gmail: ditto. Drobox: ditto. MS Windows: Ditto.

And all of these firms store data on US territory, or are US based, which basically tells me my data is being scanned.

Now, I am aware of this, but not overly comfortable with it. I like my privacy. I like the idea of being innocent until proven otherwise.

So, how about taking matters in one's own hands? How about setting up my own email and cloud services?

Services wanted

Just freewheeling here, but how about: 

• replace gmail by dovecot 
• replace dropbox by nextcloud (successor/fork of owncloud) 
• create some virtual/cloud computing platform to replace ESXi. My own Azure, so to speak. 

I do have a previous (not documented) ESXi server build, and I run some 10 virtual machines on it, one of them being FreeNAS - because of it's native ZFS.
This works as a charm, despite the FreeNAS community being.... let's say sceptical, about the idea.
The only problem is that ESXi looses connection, and that is canof hard to re-establish.
So, I felt the urge to build a dedicated, 24/7 storage server. 
This would also have to take care of some laptop storage, and runs 24/7, so it better be energy-efficient. Of course, I'm not able, budget wise, to go for the ultimate option, SSD-only. A mix of SSD and 2.5" drives should do, and I could probably salvage some 3TB disks of the ESXi build. 

Enter Sub-project 1: storage server.

OAM PS3 State-of-the-art

An attempt to run OAM 11G Release 2 PS3 on Oracle Linux 6.7, WLS 12C, RDBMS 12C.

Install Linux

Pretty straightforward. Used Oracle 6.7, as 7 is not certified. Create a 200MB /boot, and an LVM for /, both ext4. Install just the server. Deselect *all* options, just X system and X legacy support (the OUI needs it). Some 566 packages will get installed. Make sure it boots, and the network starts.

Linux Maintenance

Change /etc/sysconfig/selinux to read

SELINUX=disabled

I needed to use

ifup eth0

Address that by editing /etc/sysconfig/network-scripts/ifcfg-eth0 and change

ONBOOT=yes

Update to the latest:

yum update

Reboot...
As a pleasant surprise, my 6.6 was updated to 6.7, according to the boot messages.
Then, log in as root again, and start preparing for installs:

yum install oracle-rdbms-server-12cR1-preinstall

Also, add the following to /etc/sysctl.conf:

# IPv6 disabled net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1

As the oracle preinstall alters you grub, you do want to... reboot.

Oracle installs

[root@tvs ~]# mkdir /oracle [root@tvs ~]# chown oracle:oinstall /oracle [root@tvs ~]# passwd oracle Changing password for user oracle. New password: Retype new password: passwd: all authentication tokens updated successfully.

As oracle:

[oracle@tvs ~]$ mkdir /oracle/depot

Mount my private storage:

[root@tvs ~]# mount -o soft,intr,rsize=8192,wsize=8192,nolock 192.168.4.198:/volume2/oracle/Software /oracle/depot

Database

/oracle/depot/12G/database/runInstaller - fails with

"PRVF-0002: could not retrieve local node name".

I have to modify the hosts file (/etc/hosts), and add the current ip address and host name. I chose just to install the software, for a single instance database, changed the base to /oracle/app, and that changes the db software location to /oracle/app/product/12.1.0/dbhome_1. All else remains default.
Alter .bash_profile:

# Additional stuff export ORACLE_HOME=/oracle/app/product/12.1.0/dbhome_1 export PATH=$ORACLE_HOME/bin:$PATH export TNS_ADMIN=$ORACLE_HOME/network/admin

Java

Install the latest Java JDK; WLS 12C needs a development environment...

tar zxf /oracle/depot/weblogic/jdk-8u51-linux-x64.tar.gz -C /oracle mv /oracle/jdk1.8.0_51/* /oracle/java/

Alter .bash_profile, source it, and check:

export JAVA_HOME=/oracle/java/jre export PATH=$JAVA_HOME/bin:$PATH [oracle@tvs] . .bash_profile [oracle@tvs] java -version java version "1.8.0_51" Java(TM) SE Runtime Environment (build 1.8.0_51-b16) Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)

This version is not aligned with the installers, these make Java throw this warning:

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0

Use netca to create a listener, or manually, if you like that:

[oracle@tvs ~]$ cat /oracle/app/product/12.1.0/dbhome_1/network/admin/listener.ora # listener.ora Network Configuration File: /oracle/app/product/12.1.0/dbhome_1/network/admin/listener.ora # Generated by Oracle configuration tools. LISTENER = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = tvs)(PORT = 1521)) ) )

If you want to cut-n-paste the above code, make sure to manually start the listener, using lsnrctl start.

Repository Creation Utility

/oracle/depot/OFM/11.1.2.3.0/rcu_linux_11.1.1.9.0_64/rcuHome/bin/rcu

OK, that has no problems installing against a 12C database. I heart (never seen it, though) that previous versions would not install against 12C.

Weblogic 12C

java -jar -d64 /oracle/depot/weblogic/fmw_12.1.3.0.0_wls.jar

Do NOT start the Configuration Wizard. Also, make sure, this is the first install in your middleware home - WLS refuses to install in a non-empty location.

OUD

Has been done before, nothing new, but for the Java version used here.

/oracle/depot/OFM/11.1.2.3.0/oud_11.1.2.3.0/Disk1/runInstaller -jreLoc $JAVA_HOME

Just as well create an instance, as described here.

OAM

Has been done before, too. Refer to that blog entry.

Redo...

WLS12C does NOT work with OAM. The config.sh never comes to an end other than "Too many parameters". Besides, it's not certified according to the matrix, as is Java 8 (Java 8 is not certified, that is.) So, I guess state of the art is Linux 7 (yes - that is certified!), WLS 10.3.6.11, Java 7 update 80.

OAM PS3 - continued

Allow auto start (production mode) for your scripts:

cd /oracle/user_projects/domains/oam_domain/servers mkdir -p oam_server1/security mkdir -p omsm_server1/security mkdir -p oam_policy_mgr1/security vi oam_server1/security/boot.properties cp oam_server1/security/boot.properties omsm_server1/security/ cp oam_server1/security/boot.properties oam_policy_mgr1/security/

You can now use command line scripts to start the other servers (oam_server1, omsm_server1, and oam_policy_mgr1), like so:

/oracle/user_projects/domains/oam_domain/bin/startManagedWebLogic.sh oam_server1

Starting using WLS Console (GUI)

In the OAM 11GRel2PS2 setup, I created a "machine". This is not a physical machine, just a weblogic placeholder. I need one to allow the nodemanager to start/stop all servers.
If I don, I will get these messages trying to control (start/stop) a managed server via de admin server console:

So, login on the console, choose to expand the Environment, click the "Lock & Edit' button, and create e New Machine:

Name does not really matter, as said, it is just a place holder, and bears no connection to ant (physical) machine whatsoever. Leave OS type to Other; it is not Unix, and it is not Virtual (it is Linux).
Next, define the nodemanager. By default the nodemanager is created to listen in SSL mode, and it will listen on all addresses, so localhost should do.
In a serious clustered environment, you want to change the nodemanager to listen to the machine name or address, not just localhost - if you would, the nodemanager would not accept calls from the other cluster members.

Now, you can add the Admin, and managed servers to this machine, and control these from the GUI as well. Next task will be protecting resources.

refhost.xml kludge is fixed

No More missing packages

I wrote several times about manually editing refhost.xml. There's not need for it, just apply Patch 18231786.