Introduction
When I started this installation I configured DNS before, there are some interesting points that we have to have in mind because it could take some time to resolve them.
I will focus on the Network Prerequisites (DNS and DHCP) if you configure both of these the next process’s going to be very easy and at the end of this document I will document all the errors that I had on the installation procedure.
Points to carry out
- Configuring DNS and DHCP
- Configuring Virtual Machine
- Installing Cluster (creating DATA group in ASM)
- Checking the Configuration after the installation process
- Errors during the installation
Assumptions
You have at least 8GB of RAM
VirtualBox is installed and have enough space on your hard drive (40GB)
Configuring Virtual machine
There are several tutorials about installing OL6 on VirtualBox environment so also I create my own version you can check here if you want the virtual machine there are templates on OTN you can download those as well. You can download the Oracle Software from OTN or Software Delivery Cloud.
For this installation I just go through the easy method and just install the oracle preinstall package. Also remember to install network time protocol binaries.
yum install oracle-rdbms-server-12cR1-preinstall yum install ntp -y
After that you have to check the users and shell limits are good, for more information about this, you could check here , remember to create the necessary path for the installation process we’re going to use the OFA structure.
Checking the users
the oracle preinstall package do not create the grid user, and the other OS groups, so we’ll add them with the following:
#adding groups groupadd -g 54323 oper groupadd -g 54327 asmdba groupadd -g 54328 asmoper groupadd -g 54329 asmadmin #if you need the following groups, just uncomment and create them # groupadd -g 54324 backupdba # groupadd -g 54325 dgdba # groupadd -g 54326 kmdba #the following groups already has been created by oracle preinstall package so no actions need in case you don’t have create them # groupadd -g 54321 oinstall # groupadd -g 54322 dba #creating grid user and add oracle user to the new created groups useradd -u 502 -g oinstall -G asmoper,asmdba,asmadmin grid usermod -g oinstall -G dba,oper,asmdba oracle #setting grid and oracle password echo -e "oracle" | (passwd --stdin grid) echo -e "oracle" | (passwd --stdin oracle)
Now you can copy the oracle grid software under the grid user and oracle database software under the oracle user, I have use WinSCP to copy from my host windows machine to the Linux VM.
unzip linuxamd64_12102_grid_1of2.zip unzip linuxamd64_12102_grid_2of2.zip
Checking Resource Sheell Limits for the Oracle Software Installation Users
[root@host01 ~]# vi /etc/security/limits.conf #add the following entry: oracle soft nofile 131072 oracle hard nofile 131072 oracle soft nproc 131072 oracle hard nproc 131072 oracle soft core unlimited oracle hard core unlimited #oracle soft memlock 1048576 #oracle hard memlock 1048576 grid soft nofile 131072 grid hard nofile 131072 grid soft nproc 131072 grid hard nproc 131072 grid soft core unlimited grid hard core unlimited #grid soft memlock 1048576 #grid hard memlock 1048576 [root@host01 ~]# cat /etc/security/limits.conf oracle soft nofile 131072 oracle hard nofile 131072 oracle soft nproc 131072 oracle hard nproc 131072 oracle soft core unlimited oracle hard core unlimited #oracle soft memlock 1048576 #oracle hard memlock 1048576 grid soft nofile 131072 grid hard nofile 131072 grid soft nproc 131072 grid hard nproc 131072 grid soft core unlimited grid hard core unlimited #grid soft memlock 1048576 #grid hard memlock 1048576 # End of file
Make sure that all the paths are created properly, if not create them
mkdir -p /u01/app/12.1.0/grid mkdir -p /u01/app/grid mkdir -p /u01/app/oracle chown -R grid:oinstall /u01 chown oracle:oinstall /u01/app/oracle chmod -R 775 /u01/
Finally, just make sure that the oracle preinstall package has set the correct kernel parameters, after that shutdown the VM to proceed with the storage part.
[root@host01]# cat /etc/sysctl.conf
Now we have to add the storage part, we’ll use the shareable disk option on virtual box, in this case I am using Windows on host machine so you can create as the following:
Creating Shareable disks
D: cd D:\VirtualBox VMs\ol6u5_dns_RAC12c "c:\Program Files\Oracle\VirtualBox\VBoxManage" createhd --filename asm12cdisk1.vmdk --size 5120 --format VMDK --variant Fixed "c:\Program Files\Oracle\VirtualBox\VBoxManage" createhd --filename asm12cdisk2.vmdk --size 5120 --format VMDK --variant Fixed "c:\Program Files\Oracle\VirtualBox\VBoxManage" createhd --filename asm12cdisk3.vmdk --size 5120 --format VMDK --variant Fixed "c:\Program Files\Oracle\VirtualBox\VBoxManage" createhd --filename asm12cdisk4.vmdk --size 5120 --format VMDK --variant Fixed "c:\Program Files\Oracle\VirtualBox\VBoxManage" modifyhd asm12cdisk1.vmdk --type shareable "c:\Program Files\Oracle\VirtualBox\VBoxManage" modifyhd asm12cdisk2.vmdk --type shareable "c:\Program Files\Oracle\VirtualBox\VBoxManage" modifyhd asm12cdisk3.vmdk --type shareable "c:\Program Files\Oracle\VirtualBox\VBoxManage" modifyhd asm12cdisk4.vmdk --type shareable "c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach OL6u8_RAC_12cR1_Node1 --storagectl "SATA" --port 1 --device 0 --type hdd --medium asm12cdisk1.vmdk --mtype shareable "c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach OL6u8_RAC_12cR1_Node1 --storagectl "SATA" --port 2 --device 0 --type hdd --medium asm12cdisk2.vmdk --mtype shareable "c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach OL6u8_RAC_12cR1_Node1 --storagectl "SATA" --port 3 --device 0 --type hdd --medium asm12cdisk3.vmdk --mtype shareable "c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach OL6u8_RAC_12cR1_Node1 --storagectl "SATA" --port 4 --device 0 --type hdd --medium asm12cdisk4.vmdk --mtype shareable
Startup the virtual machine and we’ll format the new disks added, and also create the UDEV rules to ensure they will be ready to attach to ASM instance after that.
#check the current disks, in this case we have 4 raw disks. cd /dev ls /dev/sd* ++sda sda1 sda2 sdb sdc sdd sde #starting formatting the disks mentioned above, this is a very simple way to archive this, I am creating a script for this, when is ready I will share also. echo -e "n\np\n1\n\n\nw" | fdisk /dev/sdb echo -e "n\np\n1\n\n\nw" | fdisk /dev/sdc echo -e "n\np\n1\n\n\nw" | fdisk /dev/sdd echo -e "n\np\n1\n\n\nw" | fdisk /dev/sde
Configuring UDEV rules
if the file does not exist
echo -e "options=-g" > /etc/scsi_id.config
if the file already exists just make sure it has the following line if not add it “options=-g”
echo -e "options=-g" >> /etc/scsi_id.config
Getting the SCSI id of the disks
/sbin/scsi_id -g -u -d /dev/sdb /sbin/scsi_id -g -u -d /dev/sdc /sbin/scsi_id -g -u -d /dev/sdd /sbin/scsi_id -g -u -d /dev/sde # Output example: [root@host01 ~]# /sbin/scsi_id -g -u -d /dev/sdb 1ATA_VBOX_HARDDISK_VBf62ab73e-b08b2891 [root@host01 ~]# /sbin/scsi_id -g -u -d /dev/sdc 1ATA_VBOX_HARDDISK_VB70f4590e-8d03894f [root@host01 ~]# /sbin/scsi_id -g -u -d /dev/sdd 1ATA_VBOX_HARDDISK_VB41b901d7-6d96db64 [root@host01 ~]# /sbin/scsi_id -g -u -d /dev/sde 1ATA_VBOX_HARDDISK_VB0e4bc222-2af42fff
Now create or edit the file which going to have the UDEV rules
vi /etc/udev/rules.d/99-oracle-asmdevices.rules KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VBf62ab73e-b08b2891", NAME="asmdisk1p1", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB70f4590e-8d03894f", NAME="asmdisk1p2", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB41b901d7-6d96db64", NAME="asmdisk2p1", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB0e4bc222-2af42fff", NAME="asmdisk2p2", OWNER="grid", GROUP="asmadmin", MODE="0660"
Load updated block device partition tables.
/sbin/partprobe /dev/sdb1 /sbin/partprobe /dev/sdc1 /sbin/partprobe /dev/sdd1 /sbin/partprobe /dev/sde1
Test the rules are working as expected.
/sbin/udevadm test /block/sdb/sdb1
Now we have to reload the UDEV rules and after that start UDEV demon.
/sbin/udevadm control --reload-rules /sbin/start_udev
Check if the disks are visible, if not you have to check the UDEV configuration and fixed before continue with the process.
[root@host01 ~]# ls -ltr /dev/asm* brw-rw----. 1 grid asmadmin 8, 17 Jul 19 01:02 /dev/asmdisk1p1 brw-rw----. 1 grid asmadmin 8, 33 Jul 19 01:02 /dev/asmdisk1p2 brw-rw----. 1 grid asmadmin 8, 65 Jul 19 01:02 /dev/asmdisk2p2 brw-rw----. 1 grid asmadmin 8, 49 Jul 19 01:02 /dev/asmdisk2p1 [root@host01 ~]#
Now you have to update the /etc/hosts file, we are using DHCP and DNS therefore just public and private network IPs will be included in this file
Editing hosts filename
# Public (eth0) 192.168.100.101 host01.example.com host01 192.168.100.102 host02.example.com host02 192.168.100.103 host03.example.com host03 192.168.100.104 host04.example.com host04 192.168.100.105 host05.example.com host05 192.168.100.106 host06.example.com host06 # Private (eth1) 192.168.50.101 host01-priv1.example.com host01-priv1 192.168.50.102 host02-priv1.example.com host02-priv1 192.168.50.103 host03-priv1.example.com host03-priv1 192.168.50.104 host04-priv1.example.com host04-priv1 192.168.50.105 host05-priv1.example.com host05-priv1 192.168.50.106 host06-priv1.example.com host06-priv1
Setting the DNS server as following:
[root@host01 ~]# vi /etc/resolv.conf #set the following 3 lines options timeout:1 attempts:1 rotate search example.com nameserver 192.168.100.175 [root@host01 ~]$ cat /etc/resolv.conf options timeout:1 attempts:1 rotate search example.com nameserver 192.168.100.175
[Optional]In case Network manager overwrite the configuration on /etc/resolv.conf file we can disable it with the following commands (also if you have a NAT interface please disable it because it will overwrite this file).
$ chkconfig --list NetworkManager $ service NetworkManager stop $ chkconfig NetworkManager off $ service network start $ chkconfig network on
Configuring NTP
[root@host01 ~]# cat /etc/sysconfig/ntpd # Drop root to id 'ntp:ntp' by default. OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g" [root@host01 ~]# vi /etc/sysconfig/ntpd [root@host01 ~]# cat /etc/sysconfig/ntpd # Drop root to id 'ntp:ntp' by default. OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid" [root@host01 ~]# service ntpd restart Shutting down ntpd: [FAILED] Starting ntpd: [ OK ] [root@host01 ~]# chkconfig ntpd on
Make sure the SELINUX is set on permissive mode
[root@host01 ~]# cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=permissive # SELINUXTYPE= can take one of these two values: # targeted - Targeted processes are protected, # mls - Multi Level Security protection. SELINUXTYPE=targeted
Now the virtual machine template was finished we could clone it, you have two options clone just the disk and then create the another virtual machine using cloned disk, or you can detach the ASM disks, clone the complete virtual machine and then reattach the ASM disk in all the nodes.
In my case I use the last step, I will not show you here but the procedure is the same that I have explained before.
So now we have a fresh copy of the VM template, now we have to adjust some configurations to avoid problems during the cluster installation process.
Basically the steps are the following:
- Attach the ASM disks
- Check the visibility of ASM disks
- Change hostname
- Change IP address on each interface
When all the prerequisites before mentioned are okay, we can run the tool cluvfy to check if everything is set.
Installing Cluster (creating DATA group in ASM)
Now we can install the oracle Clusterware software, just go to the grid user an start the installation process running OUI.
[root@host01 ~]# xhost + access control disabled, clients can connect from any host [root@host01 ~]# su - grid [grid@host01 ~]$ cd grid [grid@host01 grid]$ pwd /home/grid/grid [grid@host01 grid]$ ./runInstaller
After that the first step will be showing to you, this step is if you want to set up your oracle account so select “skip sofwate update” and then click in the next button.
Choosing the installation and cluster type,
Select “install and configure Oracle Grid Infrastructure for a Cluster”, click in next button.
Select “Configure a Flex Cluster” and click next button
Select language after that click on next button
Grid Plug and play, this step is very important, in this specific installation we’ll use GNS to resolve SCAN virtual IPs, so we have to configuring as following:
- Cluster name: cluster01
- SCAN name: cluster01-scan.cluster01.example.com
- SCAN port: 1521
- Configure GNS: Select option
- Configure Nodes Virtual IP: select option
- Create a New GNS: select option
- GNS VIP Adress: 192.168.100.99
- GNS sub domain: cluster01.example.com
In the cluster node information, you have to add the information of all the nodes that you’re going to add. In this case we’ll add two hub nodes.
In the network interface usage, we’ll add eth0 for public and eth1 for private & ASM, you should set up SSH connectivity if is not already configured.
Create ASM disk group, remember to change the discovery path to “/dev/asm*” and select at least 3 disks
Set the ASM password instance
Failure Isolation Support, in this installation we won’t use IPMI
If you have EM configured you can set up the configuration now
Set the OS groups as following
Set the installation location as following
- Oracle home: /u01/app/grid
- Oracle base: /u01/app/12.1.0/grid
Root and script execution
You can set the root password to run automatically the inventory scripts on all the nodes in the cluster…
Prerequisite checks, in this case we are using UDEV rules for ASM disk, so you can safely ignore it.
On the summary step just check that everything is okay, and start the installation process
The final step is checking the configuration, the following can help you with this
Checking the Configuration after the installation process
Checking that CRS and CSS process are up and running on each cluster node.
[root@host01 trace]# su - grid [grid@host01 ~]$ . oraenv ORACLE_SID = [grid] ? +ASM1 The Oracle base has been set to /u01/app/grid [grid@host01 ~]$ crsctl check cluster -all ************************************************************** host01: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** host02: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online **************************************************************
Checking all the resources on the cluster, check that all are online status.
[grid@host01 ~]$ crsctl stat res -t -------------------------------------------------------------------------------- Name Target State Server State details -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.ASMNET1LSNR_ASM.lsnr ONLINE ONLINE host01 STABLE ONLINE ONLINE host02 STABLE ora.DATA.dg ONLINE ONLINE host01 STABLE ONLINE ONLINE host02 STABLE ora.LISTENER.lsnr ONLINE ONLINE host01 STABLE ONLINE ONLINE host02 STABLE ora.net1.network ONLINE ONLINE host01 STABLE ONLINE ONLINE host02 STABLE ora.ons ONLINE ONLINE host01 STABLE ONLINE ONLINE host02 STABLE -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE host02 STABLE ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE host01 STABLE ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE host01 STABLE ora.MGMTLSNR 1 ONLINE ONLINE host01 169.254.208.71 192.1 68.50.101,STABLE ora.asm 1 ONLINE ONLINE host01 Started,STABLE 2 ONLINE ONLINE host02 Started,STABLE 3 OFFLINE OFFLINE STABLE ora.cvu 1 ONLINE ONLINE host01 STABLE ora.gns 1 ONLINE ONLINE host01 STABLE ora.gns.vip 1 ONLINE ONLINE host01 STABLE ora.host01.vip 1 ONLINE ONLINE host01 STABLE ora.host02.vip 1 ONLINE ONLINE host02 STABLE ora.mgmtdb 1 ONLINE ONLINE host01 Open,STABLE ora.oc4j 1 ONLINE ONLINE host01 STABLE ora.scan1.vip 1 ONLINE ONLINE host02 STABLE ora.scan2.vip 1 ONLINE ONLINE host01 STABLE ora.scan3.vip 1 ONLINE ONLINE host01 STABLE -------------------------------------------------------------------------------- [grid@host01 ~]$
Finally, if you want to check the configuration of GNS and SCAN, go ahead and check with the following:
cheeking GNS configuration
[grid@host01 grid]$ . oraenv ORACLE_SID = [grid] ? +ASM1 The Oracle base has been set to /u01/app/grid [grid@host01 grid]$ srvctl config gns -a -l GNS is enabled. GNS is listening for DNS server requests on port 53 GNS is using port 5353 to connect to mDNS GNS status: OK Domain served by GNS: cluster01.example.com GNS version: 12.1.0.2.0 Globally unique identifier of the cluster where GNS is running: 6c2b9db098094fb2ffb6b49d140371f3 Name of the cluster where GNS is running: cluster01 Cluster type: server. GNS log level: 1. GNS listening addresses: tcp://192.168.100.99:38956. GNS is individually enabled on nodes: GNS is individually disabled on nodes:
chekcing SCAN configuration
[grid@host01 grid]$ srvctl config scan SCAN name: cluster01-scan.cluster01.example.com, Network: 1 Subnet IPv4: 192.168.100.0/255.255.255.0/eth0, dhcp Subnet IPv6: SCAN 0 IPv4 VIP: -/scan1-vip/192.168.100.129 SCAN VIP is enabled. SCAN VIP is individually enabled on nodes: SCAN VIP is individually disabled on nodes: SCAN 1 IPv4 VIP: -/scan2-vip/192.168.100.128 SCAN VIP is enabled. SCAN VIP is individually enabled on nodes: SCAN VIP is individually disabled on nodes: SCAN 2 IPv4 VIP: -/scan3-vip/192.168.100.125 SCAN VIP is enabled. SCAN VIP is individually enabled on nodes: SCAN VIP is individually disabled on nodes: [grid@host01 grid]$
Errors during the installation process
PRCR-1064 : Failed to start resource ora.LISTENER.lsnr on node host02
CRS-5017: The resource action “ora.host02.vip start” encountered the following error:
CRS-5015: Agent timed out waiting for a response from the DHCP server
. For details refer to “(:CLSN00107:)” in “/u01/app/grid/diag/crs/host02/crs/trace/crsd_orarootagent_root.trc”.
Solution
Check if your dhcp server is up and running… dhcpd command can give you a wide description about this issue. Also when the root script is running on installation process you can check /etc/var/messages to check if there is any error related to dhcp Deamon.
CLSRSC-507: The root script cannot proceed on this node host02
Solution
Check the rootcrs log, it will give you important information about the issue, in my case when I clone the virtual machine, for some reason (not sure why) the UDEV rules was not loaded, so when I run the root scritp on host02 the storage was not present.
All the information was on CRS log
[root@host01 trace]# tail -500 /u01/app/12.1.0/grid/cfgtoollogs/crsconfig/rootcrs_host01_2017-03-23_09-38-56PM.log
….
2017-03-23 22:13:04: oracle.ops.mgmt.rawdevice.OCRException: PROC-32: Cluster Ready Services on the local node is not running Messaging error [gipcretConnectionRefused] [29]
….
2017-03-23 22:13:19: CLSRSC-507: The root script cannot proceed on this node host02 because either the first-node operations have not completed on node host01 or there was an error in obtaining the status of the first-node operations.
For more information, (Doc ID 1919825.1)
An Oracle DBA Journey 
Pingback: Adding a HUB node to Flex Cluster with GNS in RAC 12cR1 using addnode.sh | An Oracle DBA Journey