Wednesday, June 01, 2011

Grid Infrastructure Startup Issues

We have a two node RAC (Oracle 11gR2) configuration running on Linux environment. This is a test environment where lot of testing happens. The second node of the two node RAC configuration was down and it refused to start. The OHASD was up but CRS, CSS, and EVMD were offline.

I started troubleshooting the issue in the following order:

1) Reading the ohasd.log and found nothing worth interesting.

2) No file permission issues on the disks were reported.

3) Voting Disks are located in ASM and were also accessible to the second node.

4) Moving ahead, following error was reported in the ocssd.log

2011-04-06 21:13:57.781: [    CSSD][1111677248]clssnmvDHBValidateNCopy: node 1, tptrac1, has a disk HB, but no network HB, DHB has rcfg 183608157, wrtcnt, 41779628, LATS 3108751988, lastSeqNo 41779625, uniqueness 1294698909, timestamp 1302104569/3108688378

The error message clearly says there is no network heart beat between the two nodes. Indeed, it failed when I tried to ping using the private IP address. On node1, “eth1” was messed up as shown below:

 
[root@tptrac1 ~]# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr F4:CE:46:84:F7:CA
          inet6 addr: fe80::f6ce:46ff:fe84:f7ca/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3244634 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6800251 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2828581056 (2.6 GiB)  TX bytes:1669807875 (1.5 GiB)
          Interrupt:162 Memory:f6000000-f6012800

[root@tptrac1 ~]#

I reassigned the private IP address as shown below:

ifconfig eth1 10.28.177.1 netmask 255.255.255.128 up

I was able to ping after setting the private IP address on Node1. It was now time to stop and start the cluster.

-bash-3.2$ ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
-bash-3.2$

Well, the cluster happily came up without reporting any errors.

I would have saved all the troubleshooting time if I had checked node reachability in the first place. Anyways, it was a good troubleshooting exercise and I also got something to share on my blog.

Wednesday, April 06, 2011

Errors when installing “Guest Additions” using Oracle VM Virtual Box & Oracle Enterprise Linux

After upgrading Oracle VM VirtualBox from 3.x to 4.0.4, I created a new virtual machine and installed Oracle Enterprise Linux 5.2 (64-bit). While installing “Guest Additions” on the guest OS, I received the following error message:

***************************************************

Building the VirtualBox Guest Additions kernel modules

The headers for the current running kernel were not found. If the following module compilation fails then this could be the reason.

The missing package can be probably installed with

yum install kernel-devel-2.6.18-92.el5 [FAILED]

***************************************************

The error description message in the “/var/log/vboxadd-install.log” log file was pretty clear. One of the RPM’s was missing.

Below is my first (failed) attempt of “Guest Additions” installation:

 [root@rac1 u01]# ./VBoxLinuxAdditions.run
Verifying archive integrity... All good.
Uncompressing VirtualBox 4.0.4 Guest Additions for Linux.........
VirtualBox Guest Additions installer
Removing installed version 4.0.4 of VirtualBox Guest Additions...
Removing existing VirtualBox DKMS kernel modules           [  OK  ]
Removing existing VirtualBox non-DKMS kernel modules       [  OK  ]
Building the VirtualBox Guest Additions kernel modules
The headers for the current running kernel were not found. If the following module compilation fails then this could be the reason.
The missing package can be probably installed with
yum install kernel-devel-2.6.18-92.el5
                                                           [FAILED]
Your system does not seem to be set up to build kernel modules.
Look at /var/log/vboxadd-install.log to find out what went wrong.
Once you have corrected it, you can run

  /etc/init.d/vboxadd setup

to build them.

Doing non-kernel setup of the Guest Additions              [  OK  ]
Installing the Window System drivers
Installing X.Org 7.1 modules                               [  OK  ]
Setting up the Window System to use the Guest Additions    [  OK  ]
You may need to restart the hal service and the Window System (or just restart
the guest system) to enable the Guest Additions.

Installing graphics libraries and desktop services componen[  OK  ]
[root@rac1 u01]# 

To get the thing working, we need to install the following RPM’s:

• kernel-devel-2.6.18-92.el5.x86_64.rpm

• kernel-headers-2.6.18-92.el5.x86_64.rpm

• glibc-headers-2.5-24.x86_64.rpm

• glibc-devel-2.5-24.x86_64.rpm

• libgomp-4.1.2-42.el5.x86_64.rpm

• gcc-4.1.2-42.el5.x86_64.rpm

I had to install 6 packages as they are interdependent.

 [root@rac1 Server]# rpm -Uvh kernel-devel-2.6.18-92.el5.x86_64.rpm 
Preparing...                ########################################### [100%]
   1:kernel-devel           ########################################### [100%]
[root@rac1 Server]# 
[root@rac1 Server]# 
[root@rac1 Server]# rpm -Uvh kernel-headers-2.6.18-92.el5.x86_64.rpm 
Preparing...                ########################################### [100%]
   1:kernel-headers         ########################################### [100%]
[root@rac1 Server]# 
[root@rac1 Server]# 
[root@rac1 Server]# rpm -Uvh glibc-headers-2.5-24.x86_64.rpm 
Preparing...                ########################################### [100%]
   1:glibc-headers          ########################################### [100%]
[root@rac1 Server]# 
[root@rac1 Server]# 
 [root@rac1 Server]# rpm -Uvh glibc-devel-2.5-24.x86_64.rpm 
Preparing...                ########################################### [100%]
   1:glibc-devel            ########################################### [100%]
[root@rac1 Server]# 
[root@rac1 Server]# 
[root@rac1 Server]# rpm -Uvh libgomp-4.1.2-42.el5.x86_64.rpm 
Preparing...                ########################################### [100%]
   1:libgomp                ########################################### [100%]
[root@rac1 Server]# 
[root@rac1 Server]# 
[root@rac1 Server]# rpm -Uvh gcc-4.1.2-42.el5.x86_64.rpm 
Preparing...                ########################################### [100%]
   1:gcc                    ########################################### [100%]
[root@rac1 Server]# 

After installing the above packages, my “Guest Additions” installation was successful.

[root@rac1 u01]# ./VBoxLinuxAdditions.run
Verifying archive integrity... All good.
Uncompressing VirtualBox 4.0.4 Guest Additions for Linux.........
VirtualBox Guest Additions installer
Removing installed version 4.0.4 of VirtualBox Guest Additions...
Removing existing VirtualBox DKMS kernel modules           [  OK  ]
Removing existing VirtualBox non-DKMS kernel modules       [  OK  ]
Building the VirtualBox Guest Additions kernel modules
Your guest system does not seem to have sufficient OpenGL support to enable
accelerated 3D effects (this requires Linux 2.6.27 or later in the guest
system).  This Guest Additions feature will be disabled.


Building the main Guest Additions module                   [  OK  ]
Building the shared folder support module                  [  OK  ]
Doing non-kernel setup of the Guest Additions              [  OK  ]
Starting the VirtualBox Guest Additions                    [  OK  ]
Installing the Window System drivers
Installing X.Org 7.1 modules                               [  OK  ]
Setting up the Window System to use the Guest Additions    [  OK  ]
You may need to restart the hal service and the Window System (or just restart
the guest system) to enable the Guest Additions.

Installing graphics libraries and desktop services componen[  OK  ]
[root@rac1 u01]# 
[root@rac1 u01]# 

I can now move the mouse freely between the host & guest OS’s, copy & paste, …

Sunday, March 27, 2011

Oracle VM VirtualBox Upgrade - Version 4.0.4

I have been running on Oracle VM VirtualBox 3.x for quite sometime. I use to ignore the upgrade dialog box and kept saying to myself that I will upgrade next time. Finally, I made my mind to upgrade to the latest and the greatest version “4.0.4”. The upgrade was pretty straight forward.

When I tried to start one of the Virtual Machines, I was greeted with the following error:

I blamed myself for upgrading the stuff but the error message was quite straight forward. I was asked to install “Oracle VM VirtualBox Extension Pack”. You can download it from here.

When you try to install a Warning pops up as shown below. Click on the “Install” button and you are done.

The next message that pops up is the success of extension pack installation.

I can now start all my VM’s. ;)