Recreate 10.2 Nodeapps,RAC Listener, ASM and DB instance after an OCR corruption – not supported !

Overview

Note: 
 - The information in this posts is not validated by Oracle, is not supported by Oracle, and should only be 
   used at your own risk; it is for educational purposes only.
 - Backup your OCR before running any commands found below
 - Most parameter are case sensitive - Database and Instances are uppercase - where hostnames are lower case - 
   be carefull with that or you can easily corrupt your OCR again 

Step 1: Cleanup OCR resources for a specific node which needs to be deleted ( ract3 )

Stop CRS stack on ract3
[root@ract3 ~]# crsctl stop crs

Remove Instance from node ract3
[oracle@ract1 ~]$ srvctl  stop instance -d RACT -i RACT3
[oracle@ract1 ~]$ srvctl remove instance -d RACT -i RACT3
Remove instance RACT3 from the database ract? (y/[n]) y

Remove ASM  from node ract3
$ srvctl stop asm -n ract3
$ srvctl remove asm -f -n ract3 

Remove Listener  from node ract3
[root@ract3 ract3]# lsnrctl stop LISTENER_RACT3 
[root@ract3 ract3]#  crs_unregister ora.ract3.LISTENER_RACT3.lsnr

Stop and remove nodeapps  from node ract3
[root@ract1 install]# srvctl stop  nodeapps -n ract3
CRS-0210: Could not find resource 'ora.ract3.LISTENER_RACT3.lsnr'.
[root@ract1 install]#  srvctl  remove  nodeapps -n ract3 
Please confirm that you intend to remove the node-level applications on node ract3 (y/[n]) y

Verify CRS resources :
[root@ract1 install]# crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....T1.inst application    ONLINE    ONLINE    ract1       
ora....T2.inst application    ONLINE    ONLINE    ract2       
ora.RACT.db    application    ONLINE    ONLINE    ract2       
ora....SM1.asm application    ONLINE    ONLINE    ract1       
ora....T1.lsnr application    ONLINE    ONLINE    ract1       
ora.ract1.gsd  application    ONLINE    ONLINE    ract1       
ora.ract1.ons  application    ONLINE    ONLINE    ract1       
ora.ract1.vip  application    ONLINE    ONLINE    ract1       
ora....SM2.asm application    ONLINE    ONLINE    ract2       
ora....T2.lsnr application    ONLINE    ONLINE    ract2       
ora.ract2.gsd  application    ONLINE    ONLINE    ract2       
ora.ract2.ons  application    ONLINE    ONLINE    ract2       
ora.ract2.vip  application    ONLINE    ONLINE    ract2     
--> All resource using ract3 are deleted now 

Verify that node ract3 is still part of our cluster
[root@ract1 install]# olsnodes -n
ract1   1
ract2   2
ract3   3

Remove node ract3 from OCR
[root@ract1 install]#  cd $ORA_CRS_HOME/install
[root@ract1 install]# ./rootdeletenode.sh ract3,3
CRS-0210: Could not find resource 'ora.ract3.LISTENER_RACT3.lsnr'.
CRS-0210: Could not find resource 'ora.ract3.ons'.
CRS-0210: Could not find resource 'ora.ract3.vip'.
CRS-0210: Could not find resource 'ora.ract3.gsd'.
CRS-0210: Could not find resource ora.ract3.vip.
CRS nodeapps are deleted successfully
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully deleted 14 values from OCR.
Key SYSTEM.css.interfaces.noderact3 marked for deletion is not there. Ignoring.
Successfully deleted 5 keys from OCR.
Node deletion operation successful.
'ract3,3' deleted successfully

[root@ract1 install]# olsnodes -n
ract1   1
ract2   2

Add resources again by manually calling commands found in rootaddnode.sh

Comands executed by rootaddnode.sh on ract1

Configure cluster topology for ract3
# /u01/app/oracle/product/crs/bin/clscfg -add -nn ract3,3 -pn ract3int,3 -hn ract3,3
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Attempting to add 1 new nodes to the configuration
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 3: ract3 ract3int ract3
Creating OCR keys for user 'root', privgrp 'root'..

Add nodeapps for node ract3
[root@ract1 install]#  /u01/app/oracle/product/crs/bin/srvctl add nodeapps -n ract3 -A ract3vip.example.com/255.255.255.0/eth1 -o /u01/app/oracle/product/crs

Verify CRS objecs for node ract3
[root@ract1 install]# crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------      
ora.ract3.gsd  application    OFFLINE   OFFLINE               
ora.ract3.ons  application    OFFLINE   OFFLINE               
ora.ract3.vip  application    OFFLINE   OFFLINE    

Restart CRS stack on node ract3
root@ract3 ract3]# crsctl start crs
root@ract3 ract3]# crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
[root@ract3 ract3]# crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------     
ora.ract3.gsd  application    ONLINE    ONLINE    ract3       
ora.ract3.ons  application    ONLINE    ONLINE    ract3       
ora.ract3.vip  application    ONLINE    ONLINE    ract3       

Configure listener on node ract3
Run netca and delete old ract3 configuration - add the listener again after deletion
[oracle@ract1 ~]$ netca &
 Select Nodes to configure : ract3 -> Delete 
 ract3...
    Deleted listener: LISTENER_RACT3
 Select Nodes to configure : ract3 -> Add 
 ract3...
 Listener configuration complete.

[oracle@ract1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------      
ora....T3.lsnr application    ONLINE    ONLINE    ract3       
ora.ract3.gsd  application    ONLINE    ONLINE    ract3       
ora.ract3.ons  application    ONLINE    ONLINE    ract3       
ora.ract3.vip  application    ONLINE    ONLINE    ract3   

Configure ASM
Retrieve the needed parameter from a working node
[oracle@ract1 ~]$ srvctl config asm -n ract1
+ASM1 /u01/app/oracle/product/10.2/rac_db1

[oracle@ract1 ~]$ srvctl add asm -n ract3  -i +ASM3  -o /u01/app/oracle/product/10.2/rac_db1
[oracle@ract1 ~]$ srvctl config asm -n ract3
+ASM3 /u01/app/oracle/product/10.2/rac_db1
[oracle@ract1 ~]$ srvctl status asm -n ract3
ASM instance +ASM3 is not running on node ract3.
[oracle@ract1 ~]$ srvctl start  asm -n ract3
[oracle@ract1 ~]$ srvctl status asm -n ract3
ASM instance +ASM3 is running on node ract3.
[oracle@ract1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....SM3.asm application    ONLINE    ONLINE    ract3       
ora....T3.lsnr application    ONLINE    ONLINE    ract3       
ora.ract3.gsd  application    ONLINE    ONLINE    ract3       
ora.ract3.ons  application    ONLINE    ONLINE    ract3       
ora.ract3.vip  application    ONLINE    ONLINE    ract3   

Configure Instance RACT3 on node ract3 
Get Database status 
[oracle@ract1 ~]$ srvctl status database -d RACT
Instance RACT1 is running on node ract1
Instance RACT2 is running on node ract2

Get configugred instances
[oracle@ract1 ~]$ srvctl config  database -d RACT
ract1 RACT1 /u01/app/oracle/product/10.2/rac_db1
ract2 RACT2 /u01/app/oracle/product/10.2/rac_db1

--> Node names are  case sensitve - at least for 10.2.0.1 !
Check how the currently running instances are configured
[oracle@ract1 ~]$   crs_stat | grep inst
NAME=ora.RACT.RACT1.inst
NAME=ora.RACT.RACT2.inst
[oracle@ract1 ~]$  crs_stat  ora.RACT.RACT1.inst
NAME=ora.RACT.RACT1.inst
TYPE=application
TARGET=ONLINE
STATE=ONLINE on ract1
--> Database ( RACT ) and Instance ( RACT1 ) are upper case - hostname ( ract1 ) are lower case 

Recreate Instance 
[oracle@ract1 ~]$ srvctl add instance -d RACT  -i RACT3 -n ract3
[oracle@ract1 ~]$ srvctl start  instance -d RACT  -i RACT3
[oracle@ract1 ~]$ srvctl status database -d RACT
Instance RACT1 is running on node ract1
Instance RACT2 is running on node ract2
Instance RACT3 is running on node ract3

[oracle@ract1 install]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------      
ora....T3.inst application    ONLINE    ONLINE    ract3         
ora....SM3.asm application    ONLINE    ONLINE    ract3       
ora....T3.lsnr application    ONLINE    ONLINE    ract3       
ora.ract3.gsd  application    ONLINE    ONLINE    ract3       
ora.ract3.ons  application    ONLINE    ONLINE    ract3       
ora.ract3.vip  application    ONLINE    ONLINE    ract3

Potential error: CRS-210 due mixing lower case and upper case names

Note we use ract3 for our instance name and ract for datbase name but we should use RACT and RACT3 
[oracle@ract1 ~]$ srvctl add instance -d ract -i ract3 -n ract3
[oracle@ract1 ~]$  srvctl status database -d RACT
Instance RACT1 is running on node ract1
Instance RACT2 is running on node ract2
PRKO-2015 : Error in checking condition of instance on node: ract3
[oracle@ract1 ~]$ srvctl start instance  -d ract -i ract3  
CRS-0210: Could not find resource ora.RACT.ract3.inst.
--> Names are case sensitve - Restore you OCR to fix ORA-210

Major commands found in rootaddnode.sh and root.sh

rootaddnode.sh on ract1 runs following major commands
# /u01/app/oracle/product/crs/bin/clscfg -add -nn ract3,3 -pn ract3int,3 -hn ract3,3
# /u01/app/oracle/product/crs/bin/srvctl add nodeapps -n ract3 -A ract3vip.example.com/255.255.255.0/eth1 -o /u01/app/oracle/product/crs
# /bin/su -l oracle -c "/u01/app/oracle/product/crs/bin/cluutil -sourcefile /etc/oracle/ocr.loc -destfile /u01/app/oracle/product/crs/srvm/admin/ocr.loc -nodelist ract3"
--> only first 2 commands are need to recreate nodeapps 

root.sh on ract3 runs following major commands
# /u01/app/oracle/product/crs/bin/crsctl create scr oracle
# /u01/app/oracle/product/crs/bin/crsctl check install -wait 600

Running vipca(silent) for configuring nodeapps
# /u01/app/oracle/product/crs/bin/vipca -silent -nodelist ract1,ract2 -nodevips ract1/ract1vip/255.255.255.0/eth1,ract2/ract2vip/255.255.255.0/eth1
--> For recreating nodeaapps none of these commands are needed

Syntax for clscfg and cluutil utilities

[root@ract1 install]# clscfg -h
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
clscfg -- Oracle cluster configuration tool
  This tool is typically invoked as part of the Oracle Cluster Ready
  Services install process. It configures cluster topology and other
  settings. Use -help for information on any of these modes.
  Use one of the following modes of operation.
  -install    - creates a new configuration
  -upgrade    - upgrades an existing configuration
  -downgrade  - downgrades an existing configuration
  -add        - adds a node to the configuration
  -delete     - deletes a node from the configuration
  -local      - creates a special single-node configuration for ASM
  -concepts   - brief listing of terminology used in the other modes
  -trace      - may be used in conjunction with any mode above for tracing
WARNING: Using this tool may corrupt your cluster configuration. Do not
         use unless you positively know what you are doing.

[root@ract1 install]# clscfg -add -h
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
clscfg -add -nn nameA,numA,nameB,numB,... -pn privA,numA,privB,numB,...
       [-hn hostA,numA,hostB,numB,...] [-t p1,p2,p3,p4]
 -nn specifies nodenames in the same fashion as -nn in -install mode
 -pn specifies private interconnect names as -pn in -install mode
 -hn specifies hostnames in the same fashion as -hn in -install mode
 -t  specifies port numbers to be used by CRS daemons on the new node(s)
     default ports: 49895,49896,49897,49898
WARNING: Using this tool may corrupt your cluster configuration. Do not
         use unless you positively know what you are doing.

[root@ract1 install]# /u01/app/oracle/product/crs/bin/cluutil -h
usage: java ClusterUtil -sourcefile  <source file location> 
     -destFile  <destination file location>  -nodelist node1[,node2,..]

Reference

Leave a Reply

Your email address will not be published. Required fields are marked *