Introduction

When I started this installation I configured DNS before, there are some interesting points that we have to have in mind because it could take some time to resolve them.

I will focus on the Network Prerequisites (DNS and DHCP) if you configure both of these the next process’s going to be very easy and at the end of this document I will document all the errors that I had on the installation procedure.

Points to carry out

• Configuring DNS and DHCP
• Configuring Virtual Machine
• Installing Cluster (creating DATA group in ASM)
• Checking the Configuration after the installation process
• Errors during the installation

Assumptions

You have at least 8GB of RAM

VirtualBox is installed and have enough space on your hard drive (40GB)

Configuring Virtual machine

There are several tutorials about installing OL6 on VirtualBox environment so also I create my own version you can check here if you want the virtual machine there are templates on OTN you can download those as well. You can download the Oracle Software from OTN or Software Delivery Cloud.

For this installation I just go through the easy method and just install the oracle preinstall package. Also remember to install network time protocol binaries.

yum install oracle-rdbms-server-12cR1-preinstall
yum install ntp -y

After that you have to check the users and shell limits are good, for more information about this, you could check here , remember to create the necessary path for the installation process we’re going to use the OFA structure.

Checking the users

the oracle preinstall package do not create the grid user, and the other OS groups, so we’ll add them with the following:

#adding groups

groupadd -g 54323 oper

groupadd -g 54327 asmdba

groupadd -g 54328 asmoper

groupadd -g 54329 asmadmin

#if you need the following groups, just uncomment and create them

# groupadd -g 54324 backupdba

# groupadd -g 54325 dgdba

# groupadd -g 54326 kmdba

#the following groups already has been created by oracle preinstall package so no actions need in case you don’t have create them

# groupadd -g 54321 oinstall

# groupadd -g 54322 dba

#creating grid user and add oracle user to the new created groups

useradd -u 502 -g oinstall -G asmoper,asmdba,asmadmin grid

usermod -g oinstall -G dba,oper,asmdba oracle

#setting grid and oracle password

echo  -e "oracle" | (passwd --stdin grid)

echo  -e "oracle" | (passwd --stdin oracle)

Now you can copy the oracle grid software under the grid user and oracle database software under the oracle user, I have use WinSCP to copy from my host windows machine to the Linux VM.

unzip linuxamd64_12102_grid_1of2.zip

unzip linuxamd64_12102_grid_2of2.zip

Checking Resource Sheell Limits for the Oracle Software Installation Users

[root@host01 ~]# vi /etc/security/limits.conf

#add the following entry:

oracle   soft   nofile    131072
oracle   hard   nofile    131072
oracle   soft   nproc    131072
oracle   hard   nproc    131072
oracle   soft   core    unlimited
oracle   hard   core    unlimited
#oracle   soft   memlock    1048576
#oracle   hard   memlock    1048576
grid   soft   nofile    131072
grid   hard   nofile    131072
grid   soft   nproc    131072
grid   hard   nproc    131072
grid   soft   core    unlimited
grid   hard   core    unlimited
#grid   soft   memlock    1048576
#grid   hard   memlock    1048576

[root@host01 ~]# cat /etc/security/limits.conf
oracle   soft   nofile    131072
oracle   hard   nofile    131072
oracle   soft   nproc    131072
oracle   hard   nproc    131072
oracle   soft   core    unlimited
oracle   hard   core    unlimited
#oracle   soft   memlock    1048576
#oracle   hard   memlock    1048576
grid   soft   nofile    131072
grid   hard   nofile    131072
grid   soft   nproc    131072
grid   hard   nproc    131072
grid   soft   core    unlimited
grid   hard   core    unlimited
#grid   soft   memlock    1048576
#grid   hard   memlock    1048576
# End of file

Make sure that all the paths are created properly, if not create them

mkdir -p  /u01/app/12.1.0/grid

mkdir -p /u01/app/grid

mkdir -p /u01/app/oracle

chown -R grid:oinstall /u01

chown oracle:oinstall /u01/app/oracle

chmod -R 775 /u01/

Finally, just make sure that the oracle preinstall package has set the correct kernel parameters, after that shutdown the VM to proceed with the storage part.

[root@host01]# cat /etc/sysctl.conf

Now we have to add the storage part, we’ll use the shareable disk option on virtual box, in this case I am using Windows on host machine so you can create as the following:

Creating Shareable disks

D:

cd D:\VirtualBox VMs\ol6u5_dns_RAC12c

"c:\Program Files\Oracle\VirtualBox\VBoxManage" createhd --filename asm12cdisk1.vmdk --size 5120 --format VMDK --variant Fixed

"c:\Program Files\Oracle\VirtualBox\VBoxManage" createhd --filename asm12cdisk2.vmdk --size 5120 --format VMDK --variant Fixed

"c:\Program Files\Oracle\VirtualBox\VBoxManage" createhd --filename asm12cdisk3.vmdk --size 5120 --format VMDK --variant Fixed

"c:\Program Files\Oracle\VirtualBox\VBoxManage" createhd --filename asm12cdisk4.vmdk --size 5120 --format VMDK --variant Fixed

"c:\Program Files\Oracle\VirtualBox\VBoxManage" modifyhd asm12cdisk1.vmdk --type shareable

"c:\Program Files\Oracle\VirtualBox\VBoxManage" modifyhd asm12cdisk2.vmdk --type shareable

"c:\Program Files\Oracle\VirtualBox\VBoxManage" modifyhd asm12cdisk3.vmdk --type shareable

"c:\Program Files\Oracle\VirtualBox\VBoxManage" modifyhd asm12cdisk4.vmdk --type shareable

"c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach OL6u8_RAC_12cR1_Node1 --storagectl "SATA" --port 1 --device 0 --type hdd --medium asm12cdisk1.vmdk --mtype shareable

"c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach OL6u8_RAC_12cR1_Node1 --storagectl "SATA" --port 2 --device 0 --type hdd --medium asm12cdisk2.vmdk --mtype shareable

"c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach OL6u8_RAC_12cR1_Node1 --storagectl "SATA" --port 3 --device 0 --type hdd --medium asm12cdisk3.vmdk --mtype shareable

"c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach OL6u8_RAC_12cR1_Node1 --storagectl "SATA" --port 4 --device 0 --type hdd --medium asm12cdisk4.vmdk --mtype shareable

Startup the virtual machine and we’ll format the new disks added, and also create the UDEV rules to ensure they will be ready to attach to ASM instance after that.

#check the current disks, in this case we have 4 raw disks.

cd /dev

ls /dev/sd*

++sda  sda1  sda2  sdb  sdc  sdd  sde

#starting formatting the disks mentioned above, this is a very simple way to archive this, I am creating a script for this, when is ready I will share also.

echo -e "n\np\n1\n\n\nw" | fdisk /dev/sdb

echo -e "n\np\n1\n\n\nw" | fdisk /dev/sdc

echo -e "n\np\n1\n\n\nw" | fdisk /dev/sdd

echo -e "n\np\n1\n\n\nw" | fdisk /dev/sde

Configuring UDEV rules

if the file does not exist

echo -e "options=-g" > /etc/scsi_id.config

if the file already exists just make sure it has the following line if not add it “options=-g”

echo -e "options=-g" >> /etc/scsi_id.config

Getting the SCSI id of the disks

/sbin/scsi_id -g -u -d /dev/sdb

/sbin/scsi_id -g -u -d /dev/sdc

/sbin/scsi_id -g -u -d /dev/sdd

/sbin/scsi_id -g -u -d /dev/sde

# Output example:

[root@host01 ~]# /sbin/scsi_id -g -u -d /dev/sdb

1ATA_VBOX_HARDDISK_VBf62ab73e-b08b2891

[root@host01 ~]# /sbin/scsi_id -g -u -d /dev/sdc

1ATA_VBOX_HARDDISK_VB70f4590e-8d03894f

[root@host01 ~]# /sbin/scsi_id -g -u -d /dev/sdd

1ATA_VBOX_HARDDISK_VB41b901d7-6d96db64

[root@host01 ~]# /sbin/scsi_id -g -u -d /dev/sde

1ATA_VBOX_HARDDISK_VB0e4bc222-2af42fff

Now create or edit the file which going to have the UDEV rules

vi /etc/udev/rules.d/99-oracle-asmdevices.rules

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VBf62ab73e-b08b2891", NAME="asmdisk1p1", OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB70f4590e-8d03894f", NAME="asmdisk1p2", OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB41b901d7-6d96db64", NAME="asmdisk2p1", OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB0e4bc222-2af42fff", NAME="asmdisk2p2", OWNER="grid", GROUP="asmadmin", MODE="0660"

Load updated block device partition tables.

/sbin/partprobe /dev/sdb1

/sbin/partprobe /dev/sdc1

/sbin/partprobe /dev/sdd1

/sbin/partprobe /dev/sde1

Test the rules are working as expected.

/sbin/udevadm test /block/sdb/sdb1

Now we have to reload the UDEV rules and after that start UDEV demon.

/sbin/udevadm control --reload-rules

/sbin/start_udev

Check if the disks are visible, if not you have to check the UDEV configuration and fixed before continue with the process.

[root@host01 ~]# ls -ltr /dev/asm*

brw-rw----. 1 grid asmadmin 8, 17 Jul 19 01:02 /dev/asmdisk1p1

brw-rw----. 1 grid asmadmin 8, 33 Jul 19 01:02 /dev/asmdisk1p2

brw-rw----. 1 grid asmadmin 8, 65 Jul 19 01:02 /dev/asmdisk2p2

brw-rw----. 1 grid asmadmin 8, 49 Jul 19 01:02 /dev/asmdisk2p1

[root@host01 ~]#

Now you have to update the /etc/hosts file, we are using DHCP and DNS therefore just public and private network IPs will be included in this file

Editing hosts filename

# Public (eth0)
192.168.100.101   host01.example.com        host01
192.168.100.102   host02.example.com        host02
192.168.100.103   host03.example.com        host03
192.168.100.104   host04.example.com        host04
192.168.100.105   host05.example.com        host05
192.168.100.106   host06.example.com        host06
# Private (eth1)
192.168.50.101   host01-priv1.example.com   host01-priv1
192.168.50.102   host02-priv1.example.com   host02-priv1
192.168.50.103   host03-priv1.example.com   host03-priv1
192.168.50.104   host04-priv1.example.com   host04-priv1
192.168.50.105   host05-priv1.example.com   host05-priv1
192.168.50.106   host06-priv1.example.com   host06-priv1

Setting the DNS server as following:

[root@host01 ~]# vi /etc/resolv.conf

#set the following 3 lines

options timeout:1 attempts:1 rotate

search example.com

nameserver 192.168.100.175

[root@host01 ~]$ cat /etc/resolv.conf

options timeout:1 attempts:1 rotate

search example.com

nameserver 192.168.100.175

[Optional]In case Network manager overwrite the configuration on /etc/resolv.conf file we can disable it with the following commands (also if you have a NAT interface please disable it because it will overwrite this file).

$ chkconfig --list NetworkManager

$ service NetworkManager stop

$ chkconfig NetworkManager off

$ service network start

$ chkconfig network on

Configuring NTP

[root@host01 ~]# cat /etc/sysconfig/ntpd

# Drop root to id 'ntp:ntp' by default.

OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g"

[root@host01 ~]# vi /etc/sysconfig/ntpd

[root@host01 ~]# cat /etc/sysconfig/ntpd

# Drop root to id 'ntp:ntp' by default.

OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"

[root@host01 ~]# service ntpd restart

Shutting down ntpd:                                        [FAILED]

Starting ntpd:                                             [  OK  ]

[root@host01 ~]# chkconfig ntpd on

Make sure the SELINUX is set on permissive mode

[root@host01 ~]# cat /etc/selinux/config

# This file controls the state of SELinux on the system.

# SELINUX= can take one of these three values:

#     enforcing - SELinux security policy is enforced.

#     permissive - SELinux prints warnings instead of enforcing.

#     disabled - No SELinux policy is loaded.

SELINUX=permissive

# SELINUXTYPE= can take one of these two values:

#     targeted - Targeted processes are protected,

#     mls - Multi Level Security protection.

SELINUXTYPE=targeted

Now the virtual machine template was finished we could clone it, you have two options clone just the disk and then create the another virtual machine using cloned disk, or you can detach the ASM disks, clone the complete virtual machine and then reattach the ASM disk in all the nodes.

In my case I use the last step, I will not show you here but the procedure is the same that I have explained before.

So now we have a fresh copy of the VM template, now we have to adjust some configurations to avoid problems during the cluster installation process.

Basically the steps are the following:

• Attach the ASM disks
• Check the visibility of ASM disks
• Change hostname
• Change IP address on each interface

When all the prerequisites before mentioned are okay, we can run the tool cluvfy to check if everything is set.

Installing Cluster (creating DATA group in ASM)

Now we can install the oracle Clusterware software, just go to the grid user an start the installation process running OUI.

[root@host01 ~]# xhost +
access control disabled, clients can connect from any host
[root@host01 ~]# su - grid
[grid@host01 ~]$ cd grid
[grid@host01 grid]$ pwd
/home/grid/grid
[grid@host01 grid]$ ./runInstaller

After that the first step will be showing to you, this step is if you want to set up your oracle account so select “skip sofwate update” and then click in the next button.

Choosing the installation and cluster type,

Select “install and configure Oracle Grid Infrastructure for a Cluster”, click in next button.

Select “Configure a Flex Cluster” and click next button

Select language after that click on next button

Grid Plug and play, this step is very important, in this specific installation we’ll use GNS to resolve SCAN virtual IPs, so we have to configuring as following:

• Cluster name: cluster01
• SCAN name: cluster01-scan.cluster01.example.com
• SCAN port: 1521
• Configure GNS: Select option
• Configure Nodes Virtual IP: select option
• Create a New GNS: select option
• GNS VIP Adress: 192.168.100.99
• GNS sub domain: cluster01.example.com

In the cluster node information, you have to add the information of all the nodes that you’re going to add. In this case we’ll add two hub nodes.

In the network interface usage, we’ll add eth0 for public and eth1 for private & ASM, you should set up SSH connectivity if is not already configured.

Create ASM disk group, remember to change the discovery path to “/dev/asm*” and select at least 3 disks

Set the ASM password instance

Failure Isolation Support, in this installation we won’t use IPMI

If you have EM configured you can set up the configuration now

Set the OS groups as following

Set the installation location as following

• Oracle home: /u01/app/grid
• Oracle base: /u01/app/12.1.0/grid

Root and script execution

You can set the root password to run automatically the inventory scripts on all the nodes in the cluster…

Prerequisite checks, in this case we are using UDEV rules for ASM disk, so you can safely ignore it.

On the summary step just check that everything is okay, and start the installation process

The final step is checking the configuration, the following can help you with this

Checking the Configuration after the installation process

Checking that CRS and CSS process are up and running on each cluster node.

[root@host01 trace]# su - grid
[grid@host01 ~]$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/grid
[grid@host01 ~]$ crsctl check cluster -all
**************************************************************
host01:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
host02:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

Checking all the resources on the cluster, check that all are online status.

[grid@host01 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE host01 STABLE
ONLINE ONLINE host02 STABLE
ora.DATA.dg
ONLINE ONLINE host01 STABLE
ONLINE ONLINE host02 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE host01 STABLE
ONLINE ONLINE host02 STABLE
ora.net1.network
ONLINE ONLINE host01 STABLE
ONLINE ONLINE host02 STABLE
ora.ons
ONLINE ONLINE host01 STABLE
ONLINE ONLINE host02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE host02 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE host01 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE host01 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE host01 169.254.208.71 192.1
68.50.101,STABLE
ora.asm
1 ONLINE ONLINE host01 Started,STABLE
2 ONLINE ONLINE host02 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE host01 STABLE
ora.gns
1 ONLINE ONLINE host01 STABLE
ora.gns.vip
1 ONLINE ONLINE host01 STABLE
ora.host01.vip
1 ONLINE ONLINE host01 STABLE
ora.host02.vip
1 ONLINE ONLINE host02 STABLE
ora.mgmtdb
1 ONLINE ONLINE host01 Open,STABLE
ora.oc4j
1 ONLINE ONLINE host01 STABLE
ora.scan1.vip
1 ONLINE ONLINE host02 STABLE
ora.scan2.vip
1 ONLINE ONLINE host01 STABLE
ora.scan3.vip
1 ONLINE ONLINE host01 STABLE
--------------------------------------------------------------------------------
[grid@host01 ~]$

Finally, if you want to check the configuration of GNS and SCAN, go ahead and check with the following:

cheeking GNS configuration

[grid@host01 grid]$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/grid
[grid@host01 grid]$ srvctl config gns -a -l
GNS is enabled.
GNS is listening for DNS server requests on port 53
GNS is using port 5353 to connect to mDNS
GNS status: OK
Domain served by GNS: cluster01.example.com
GNS version: 12.1.0.2.0
Globally unique identifier of the cluster where GNS is running: 6c2b9db098094fb2ffb6b49d140371f3
Name of the cluster where GNS is running: cluster01
Cluster type: server.
GNS log level: 1.
GNS listening addresses: tcp://192.168.100.99:38956.
GNS is individually enabled on nodes:
GNS is individually disabled on nodes:

chekcing SCAN configuration

[grid@host01 grid]$ srvctl config scan
SCAN name: cluster01-scan.cluster01.example.com, Network: 1
Subnet IPv4: 192.168.100.0/255.255.255.0/eth0, dhcp
Subnet IPv6:
SCAN 0 IPv4 VIP: -/scan1-vip/192.168.100.129
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
SCAN 1 IPv4 VIP: -/scan2-vip/192.168.100.128
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
SCAN 2 IPv4 VIP: -/scan3-vip/192.168.100.125
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
[grid@host01 grid]$

 

Errors during the installation process

PRCR-1064 : Failed to start resource ora.LISTENER.lsnr on node host02
CRS-5017: The resource action “ora.host02.vip start” encountered the following error:
CRS-5015: Agent timed out waiting for a response from the DHCP server
. For details refer to “(:CLSN00107:)” in “/u01/app/grid/diag/crs/host02/crs/trace/crsd_orarootagent_root.trc”.

Solution

Check if your dhcp server is up and running… dhcpd command can give you a wide description about this issue. Also when the root script is running on installation process you can check /etc/var/messages to check if there is any error related to dhcp Deamon.

CLSRSC-507: The root script cannot proceed on this node host02

Solution

Check the rootcrs log, it will give you important information about the issue, in my case when I clone the virtual machine, for some reason (not sure why) the UDEV rules was not loaded, so when I run the root scritp on host02 the storage was not present.

All the information was on CRS log

[root@host01 trace]# tail -500 /u01/app/12.1.0/grid/cfgtoollogs/crsconfig/rootcrs_host01_2017-03-23_09-38-56PM.log
….
2017-03-23 22:13:04: oracle.ops.mgmt.rawdevice.OCRException: PROC-32: Cluster Ready Services on the local node is not running Messaging error [gipcretConnectionRefused] [29]
….
2017-03-23 22:13:19: CLSRSC-507: The root script cannot proceed on this node host02 because either the first-node operations have not completed on node host01 or there was an error in obtaining the status of the first-node operations.
For more information, (Doc ID 1919825.1)