Add a Node to 10.2.0.1 RAC using addNote.sh

Overview

To add node to 10.2 RAC database is done by using following steps

  • Verify OS settings for the new Node ract3
  • Run ./addNode.sh on ract1  from $ORA_CRS_HOME to install CW sofware and verify CW Installation
  • Run ./addNode.sh ract1  from $ORACLE_HOME to install RDBMS sofware
  • Use netca on cluster node ract1 and configure ract3 listener
  • Use dbca on ract1 and finally add ASM and Rdbms Instance on ract3

Note OUI  writes addNode.sh logs to   /home/oracle/oraInventory/logs/

OS Verification for node ract3

Verify DNS
[root@ract3 ~]# nslookup ract3
Name:   ract3.example.com
Address: 192.168.1.133

# nslookup ract3int
Name:   ract3int.example.com
Address: 192.168.2.133

Reverse Address resolution :  
# nslookup  192.168.1.133
133.1.168.192.in-addr.arpa      name = ract3.example.com.

# nslookup 192.168.2.133
133.2.168.192.in-addr.arpa      name = ract3int.example.com.

Verify Network devices
#  ifconfig | egrep 'eth|inet addr'
eth0      Link encap:Ethernet  HWaddr 08:00:27:CE:53:81  
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
eth1      Link encap:Ethernet  HWaddr 08:00:27:F4:CF:0C  
          inet addr:192.168.1.133  Bcast:192.168.1.255  Mask:255.255.255.0
eth2      Link encap:Ethernet  HWaddr 08:00:27:CA:C8:09  
          inet addr:192.168.2.133  Bcast:192.168.2.255  Mask:255.255.255.0
          inet addr:127.0.0.1  Mask:255.0.0.0

Verify RAW devices 
#  raw -qa
/dev/raw/raw1:  bound to major 8, minor 17
/dev/raw/raw2:  bound to major 8, minor 33
/dev/raw/raw3:  bound to major 8, minor 49
/dev/raw/raw4:  bound to major 8, minor 65
/dev/raw/raw5:  bound to major 8, minor 81
#  ls -l  /dev/raw/ra*
crw-r----- 1 root   oinstall 162, 1 Apr  9 13:24 /dev/raw/raw1
crw-r----- 1 root   oinstall 162, 2 Apr  9 13:24 /dev/raw/raw2
crw-r--r-- 1 oracle oinstall 162, 3 Apr  9 13:24 /dev/raw/raw3
crw-r--r-- 1 oracle oinstall 162, 4 Apr  9 13:24 /dev/raw/raw4
crw-r--r-- 1 oracle oinstall 162, 5 Apr  9 13:24 /dev/raw/raw5

Verify ASM disks
# /usr/sbin/oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
# /usr/sbin/oracleasm listdisks
ASM_DATA01
ASM_DATA02
ASM_DATA03
ASM_DATA04

Verify SSH connectivity
[root@ract3 ~]# su - oracle
[oracle@ract3 ~]$ ssh ract1 date
Wed Apr  9 13:26:46 CEST 2014
[oracle@ract3 ~]$ ssh ract2 date
Wed Apr  9 13:26:55 CEST 2014

Run cluvfy: 
[oracle@ract3 cluvfy12]$   ./bin/cluvfy comp sys -p crs -r 10gR2 -n ract3
Verifying system requirement 
Total memory check passed
Available memory check passed
Swap space check passed
Free disk space check passed for "ract3:/tmp"
Check for multiple users with UID value 500 passed 
User existence check passed for "oracle"
Check for multiple users with UID value 99 passed 
User existence check passed for "nobody"
Group existence check passed for "oinstall"
Group existence check passed for "dba"
Membership check for user "oracle" in group "oinstall" [as Primary] passed
Membership check for user "oracle" in group "dba" passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed
Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Package existence check passed for "make"
Package existence check passed for "binutils"

WARNING: 
PRVF-7584 : Multiple versions of package "control-center" found on node ract3: control-center(x86_64)-2.16.0-16.el5,control-center(i386)-2.16.0-16.el5
Package existence check passed for "control-center"
Package existence check passed for "gcc"

WARNING: 
PRVF-7584 : Multiple versions of package "libstdc++" found on node ract3: libstdc++(x86_64)-4.1.2-54.el5,libstdc++(i386)-4.1.2-54.el5
Package existence check passed for "libstdc++"
Package existence check passed for "libstdc++-devel"
Package existence check passed for "sysstat"
Package existence check passed for "setarch"

WARNING: 
PRVF-7584 : Multiple versions of package "glibc" found on node ract3: glibc(x86_64)-2.5-118.el5_10.2,glibc(i686)-2.5-118.el5_10.2
Package existence check passed for "glibc"
Package existence check passed for "glibc-common"
Check for multiple users with UID value 0 passed 

Starting check for consistency of primary group of root user

Check for consistency of root user's primary group passed
Time zone consistency check passed

Verification of system requirement was successful. 

If getting fixable errors run cluvfy with -verbose -fixup  
[oracle@ract3 ract3]$  ./bin/cluvfy comp sys -p crs -r 10gR2 -n ract3 -verbose -fixup

Now check CRS readiness with cluvfy
[oracle@ract3 cluvfy12]$ ./bin/cluvfy stage -pre crsinst -n  ract3 -r 10gR2
...
Check for consistency of root user's primary group passed
Time zone consistency check passed
Verification of system requirement was successful.

Ignore potential  Error PRVG-5745  for cluvfy stage -pre nodeadd 
[oracle@ract3 cluvfy12]$ ./bin/cluvfy stage -pre nodeadd -n ract3 
Performing pre-checks for node addition 
ERROR: 
PRVG-5745 : CRS Configuration detected, Restart configuration check not valid in this environment
Verification cannot proceed
Pre-check for node addition was unsuccessful on all the nodes.
---> Run cluvfy with nodeadd option not from the node to be added ( ract3 ) - just run the command from an alreday installed node ( ract2 )
     [oracle@ract2 ~]$   cluvfy stage -pre nodeadd -n ract3 
For details check following Bugs 
Bug 12705949 : CLUVFY COMP NODECON FAILING WHEN THERE IS NO CRS INSTALLED
Bug 13343726 : LNX64-12.1-CVU: MISLEADING PRVG-5745 MESSAGE BEFORE CRS FRESH INSTALLATION

Install CRS software

Start AddNode.sh script from an already installed RAC instance
First testing ssh connectivity  to our newly connecting node
[oracle@ract1 ~]$ ssh ract3 date
Wed Apr  9 14:38:27 CEST 2014
[oracle@ract1 bin]$ cd /u01/app/oracle/product/crs/oui/bin
[oracle@ract1 bin]$ ./addNode.sh 
Starting Oracle Universal Installer...
Select Node                 : ract3  
       Cluster-Interconnect : ract3int  
       VIP                  : ract3vip

Run rootaddnode.sh on ract1:
[root@ract1 etc]# /u01/app/oracle/product/crs/install/rootaddnode.sh
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Attempting to add 1 new nodes to the configuration
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 3: ract3 ract3int ract3
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Failure -2 opening file handle for (raw5)
Failed to update the voting device raw5 with addnode info. 1
/u01/app/oracle/product/crs/bin/srvctl add nodeapps -n ract3 -A ract3vip.example.com/255.255.255.0/eth1 -o /u01/app/oracle/product/crs

Check OCR after running rootaddnode.sh on ract1
[root@ract1 ~]# crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------      
ora.ract3.gsd  application    OFFLINE   OFFLINE               
ora.ract3.ons  application    OFFLINE   OFFLINE               
ora.ract3.vip  application    OFFLINE   OFFLINE  

Now run root.sh on ract3
[root@ract3 ~]# /u01/app/oracle/product/crs/root.sh
WARNING: directory '/u01/app/oracle/product' is not owned by root
WARNING: directory '/u01/app/oracle' is not owned by root
WARNING: directory '/u01/app' is not owned by root
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured
Oracle CRS stack is already configured and will be running under init(1M)

[root@ract3 ~] crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

[root@ract3 ~]# crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.ract3.gsd  application    ONLINE    ONLINE    ract3       
ora.ract3.ons  application    ONLINE    ONLINE    ract3       
ora.ract3.vip  application    ONLINE    ONLINE    ract   

--> CRS is  up and running - need to add listener and ASM and RAC instances

Install database software

[oracle@ract1 ~]$ cd $ORACLE_HOME/oui/bin/
[oracle@ract1 bin]$ ./addNode.sh
Starting Oracle Universal Installer...
--> Select new node  : ract3

Configure listener on node ract3

Start netca on ract1
$ netca &
Select the Type of Oracle Net Services Configuration     Select Cluster configuration
Select the nodes to configure                            Only select the new Oracle RAC node: ract3
Type of Configuration                                    Select Listener configuration.
Listener Configuration
Next 6 Screens                                           The following screens are now like any other normal listener configuration. 
                                                         What do you want to do: Add
                                                         Listener name: LISTENER
                                                         Selected protocols: TCP
                                                         Port number: 1521
                                                         Configure another listener: No
                                                         Listener configuration complete! [ Next ]

Verify listener configuration 
[oracle@ract3 rac_db1]$ crs_stat ora.ract3.LISTENER_RACT3.lsnr
NAME=ora.ract3.LISTENER_RACT3.lsnr
TYPE=application
TARGET=ONLINE
STATE=ONLINE on ract3

[oracle@ract3 rac_db1]$  lsnrctl status LISTENER_RACT3
LSNRCTL for Linux: Version 10.2.0.1.0 - Production on 09-APR-2014 18:37:45
Copyright (c) 1991, 2005, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ract3vip.example.com)(PORT=1521)(IP=FIRST)))
..
Service "RACT" has 2 instance(s).
  Instance "RACT1", status READY, has 1 handler(s) for this service...
  Instance "RACT2", status READY, has 1 handler(s) for this service...
..
[oracle@ract3 admin]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------      
ora....T3.lsnr application    ONLINE    ONLINE    ract3       
ora.ract3.gsd  application    ONLINE    ONLINE    ract3       
ora.ract3.ons  application    ONLINE    ONLINE    ract3       
ora.ract3.vip  application    ONLINE    ONLINE    ract3

Run dbca to configure  ASM and RDBMS  instance

 
$ dbca
Welcome Screen     Select Oracle Real Application Clusters database.
Operations     Select Instance Management.
Instance Management     Select Add an instance. 
... 
DBCA verifies the new node ract3, and as the database is configured to use ASM, prompts with the message 
   “ASM is present on the cluster but needs to be extended to the following nodes: [ract3]. 
   Do you want ASM to be extended?” 
Click on Yes to add ASM to the new instance. 

[oracle@ract3 admin]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------       
ora....T3.inst application    ONLINE    ONLINE    ract3            
ora....SM3.asm application    ONLINE    ONLINE    ract3       
ora....T3.lsnr application    ONLINE    ONLINE    ract3       
ora.ract3.gsd  application    ONLINE    ONLINE    ract3       
ora.ract3.ons  application    ONLINE    ONLINE    ract3       
ora.ract3.vip  application    ONLINE    ONLINE    ract3   

SQL> select inst_id, instance_name, host_name,  status, to_char(startup_time, 'DD-MON-YYYY HH24:MI:SS')
       from gv$instance order by inst_id;
   INST_ID INSTANCE_NAME    HOST_NAME            STATUS       TO_CHAR(STARTUP_TIME
---------- ---------------- -------------------- ------------ --------------------
         1 RACT1            ract1.example.com    OPEN         09-APR-2014 14:06:17
         2 RACT2            ract2.example.com    OPEN         09-APR-2014 10:38:20
         3 RACT3            ract3.example.com    OPEN         09-APR-2014 18:41:52

 

Potential CRS-Error

CRS-0223: Resource 'ora.ract3.gsd' has placement error.
[root@ract3 client]# crs_start ora.ract3.gsd 
CRS-1028: Dependency analysis failed because of:
'Resource in UNKNOWN state: ora.ract3.gsd'
CRS-0223: Resource 'ora.ract3.gsd' has placement error.
Fix : Try to manually restart the resource 
[root@ract3 client]#  crs_stop  ora.ract3.gsd 
Attempting to stop `ora.ract3.gsd` on member `ract3`
Stop of `ora.ract3.gsd` on member `ract3` succeeded.
[root@ract3 client]# crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.ract3.gsd  application    ONLINE    ONLINE    ract3  

CRS-210 running rootaddnode.sh running  srvctl add nodeapps - for details read following note.

Reference

 

Leave a Reply

Your email address will not be published. Required fields are marked *