Overview
Note:
- The information in this posts is not validated by Oracle, is not supported by Oracle, and should only be
used at your own risk; it is for educational purposes only.
- Backup your OCR before running any commands found below
- Most parameter are case sensitive - Database and Instances are uppercase - where hostnames are lower case -
be carefull with that or you can easily corrupt your OCR again
Step 1: Cleanup OCR resources for a specific node which needs to be deleted ( ract3 )
Stop CRS stack on ract3
[root@ract3 ~]# crsctl stop crs
Remove Instance from node ract3
[oracle@ract1 ~]$ srvctl stop instance -d RACT -i RACT3
[oracle@ract1 ~]$ srvctl remove instance -d RACT -i RACT3
Remove instance RACT3 from the database ract? (y/[n]) y
Remove ASM from node ract3
$ srvctl stop asm -n ract3
$ srvctl remove asm -f -n ract3
Remove Listener from node ract3
[root@ract3 ract3]# lsnrctl stop LISTENER_RACT3
[root@ract3 ract3]# crs_unregister ora.ract3.LISTENER_RACT3.lsnr
Stop and remove nodeapps from node ract3
[root@ract1 install]# srvctl stop nodeapps -n ract3
CRS-0210: Could not find resource 'ora.ract3.LISTENER_RACT3.lsnr'.
[root@ract1 install]# srvctl remove nodeapps -n ract3
Please confirm that you intend to remove the node-level applications on node ract3 (y/[n]) y
Verify CRS resources :
[root@ract1 install]# crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....T1.inst application ONLINE ONLINE ract1
ora....T2.inst application ONLINE ONLINE ract2
ora.RACT.db application ONLINE ONLINE ract2
ora....SM1.asm application ONLINE ONLINE ract1
ora....T1.lsnr application ONLINE ONLINE ract1
ora.ract1.gsd application ONLINE ONLINE ract1
ora.ract1.ons application ONLINE ONLINE ract1
ora.ract1.vip application ONLINE ONLINE ract1
ora....SM2.asm application ONLINE ONLINE ract2
ora....T2.lsnr application ONLINE ONLINE ract2
ora.ract2.gsd application ONLINE ONLINE ract2
ora.ract2.ons application ONLINE ONLINE ract2
ora.ract2.vip application ONLINE ONLINE ract2
--> All resource using ract3 are deleted now
Verify that node ract3 is still part of our cluster
[root@ract1 install]# olsnodes -n
ract1 1
ract2 2
ract3 3
Remove node ract3 from OCR
[root@ract1 install]# cd $ORA_CRS_HOME/install
[root@ract1 install]# ./rootdeletenode.sh ract3,3
CRS-0210: Could not find resource 'ora.ract3.LISTENER_RACT3.lsnr'.
CRS-0210: Could not find resource 'ora.ract3.ons'.
CRS-0210: Could not find resource 'ora.ract3.vip'.
CRS-0210: Could not find resource 'ora.ract3.gsd'.
CRS-0210: Could not find resource ora.ract3.vip.
CRS nodeapps are deleted successfully
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully deleted 14 values from OCR.
Key SYSTEM.css.interfaces.noderact3 marked for deletion is not there. Ignoring.
Successfully deleted 5 keys from OCR.
Node deletion operation successful.
'ract3,3' deleted successfully
[root@ract1 install]# olsnodes -n
ract1 1
ract2 2
Add resources again by manually calling commands found in rootaddnode.sh
Comands executed by rootaddnode.sh on ract1
Configure cluster topology for ract3
# /u01/app/oracle/product/crs/bin/clscfg -add -nn ract3,3 -pn ract3int,3 -hn ract3,3
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Attempting to add 1 new nodes to the configuration
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 3: ract3 ract3int ract3
Creating OCR keys for user 'root', privgrp 'root'..
Add nodeapps for node ract3
[root@ract1 install]# /u01/app/oracle/product/crs/bin/srvctl add nodeapps -n ract3 -A ract3vip.example.com/255.255.255.0/eth1 -o /u01/app/oracle/product/crs
Verify CRS objecs for node ract3
[root@ract1 install]# crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.ract3.gsd application OFFLINE OFFLINE
ora.ract3.ons application OFFLINE OFFLINE
ora.ract3.vip application OFFLINE OFFLINE
Restart CRS stack on node ract3
root@ract3 ract3]# crsctl start crs
root@ract3 ract3]# crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
[root@ract3 ract3]# crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.ract3.gsd application ONLINE ONLINE ract3
ora.ract3.ons application ONLINE ONLINE ract3
ora.ract3.vip application ONLINE ONLINE ract3
Configure listener on node ract3
Run netca and delete old ract3 configuration - add the listener again after deletion
[oracle@ract1 ~]$ netca &
Select Nodes to configure : ract3 -> Delete
ract3...
Deleted listener: LISTENER_RACT3
Select Nodes to configure : ract3 -> Add
ract3...
Listener configuration complete.
[oracle@ract1 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....T3.lsnr application ONLINE ONLINE ract3
ora.ract3.gsd application ONLINE ONLINE ract3
ora.ract3.ons application ONLINE ONLINE ract3
ora.ract3.vip application ONLINE ONLINE ract3
Configure ASM
Retrieve the needed parameter from a working node
[oracle@ract1 ~]$ srvctl config asm -n ract1
+ASM1 /u01/app/oracle/product/10.2/rac_db1
[oracle@ract1 ~]$ srvctl add asm -n ract3 -i +ASM3 -o /u01/app/oracle/product/10.2/rac_db1
[oracle@ract1 ~]$ srvctl config asm -n ract3
+ASM3 /u01/app/oracle/product/10.2/rac_db1
[oracle@ract1 ~]$ srvctl status asm -n ract3
ASM instance +ASM3 is not running on node ract3.
[oracle@ract1 ~]$ srvctl start asm -n ract3
[oracle@ract1 ~]$ srvctl status asm -n ract3
ASM instance +ASM3 is running on node ract3.
[oracle@ract1 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM3.asm application ONLINE ONLINE ract3
ora....T3.lsnr application ONLINE ONLINE ract3
ora.ract3.gsd application ONLINE ONLINE ract3
ora.ract3.ons application ONLINE ONLINE ract3
ora.ract3.vip application ONLINE ONLINE ract3
Configure Instance RACT3 on node ract3
Get Database status
[oracle@ract1 ~]$ srvctl status database -d RACT
Instance RACT1 is running on node ract1
Instance RACT2 is running on node ract2
Get configugred instances
[oracle@ract1 ~]$ srvctl config database -d RACT
ract1 RACT1 /u01/app/oracle/product/10.2/rac_db1
ract2 RACT2 /u01/app/oracle/product/10.2/rac_db1
--> Node names are case sensitve - at least for 10.2.0.1 !
Check how the currently running instances are configured
[oracle@ract1 ~]$ crs_stat | grep inst
NAME=ora.RACT.RACT1.inst
NAME=ora.RACT.RACT2.inst
[oracle@ract1 ~]$ crs_stat ora.RACT.RACT1.inst
NAME=ora.RACT.RACT1.inst
TYPE=application
TARGET=ONLINE
STATE=ONLINE on ract1
--> Database ( RACT ) and Instance ( RACT1 ) are upper case - hostname ( ract1 ) are lower case
Recreate Instance
[oracle@ract1 ~]$ srvctl add instance -d RACT -i RACT3 -n ract3
[oracle@ract1 ~]$ srvctl start instance -d RACT -i RACT3
[oracle@ract1 ~]$ srvctl status database -d RACT
Instance RACT1 is running on node ract1
Instance RACT2 is running on node ract2
Instance RACT3 is running on node ract3
[oracle@ract1 install]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....T3.inst application ONLINE ONLINE ract3
ora....SM3.asm application ONLINE ONLINE ract3
ora....T3.lsnr application ONLINE ONLINE ract3
ora.ract3.gsd application ONLINE ONLINE ract3
ora.ract3.ons application ONLINE ONLINE ract3
ora.ract3.vip application ONLINE ONLINE ract3
Potential error: CRS-210 due mixing lower case and upper case names
Note we use ract3 for our instance name and ract for datbase name but we should use RACT and RACT3
[oracle@ract1 ~]$ srvctl add instance -d ract -i ract3 -n ract3
[oracle@ract1 ~]$ srvctl status database -d RACT
Instance RACT1 is running on node ract1
Instance RACT2 is running on node ract2
PRKO-2015 : Error in checking condition of instance on node: ract3
[oracle@ract1 ~]$ srvctl start instance -d ract -i ract3
CRS-0210: Could not find resource ora.RACT.ract3.inst.
--> Names are case sensitve - Restore you OCR to fix ORA-210
Major commands found in rootaddnode.sh and root.sh
rootaddnode.sh on ract1 runs following major commands
# /u01/app/oracle/product/crs/bin/clscfg -add -nn ract3,3 -pn ract3int,3 -hn ract3,3
# /u01/app/oracle/product/crs/bin/srvctl add nodeapps -n ract3 -A ract3vip.example.com/255.255.255.0/eth1 -o /u01/app/oracle/product/crs
# /bin/su -l oracle -c "/u01/app/oracle/product/crs/bin/cluutil -sourcefile /etc/oracle/ocr.loc -destfile /u01/app/oracle/product/crs/srvm/admin/ocr.loc -nodelist ract3"
--> only first 2 commands are need to recreate nodeapps
root.sh on ract3 runs following major commands
# /u01/app/oracle/product/crs/bin/crsctl create scr oracle
# /u01/app/oracle/product/crs/bin/crsctl check install -wait 600
Running vipca(silent) for configuring nodeapps
# /u01/app/oracle/product/crs/bin/vipca -silent -nodelist ract1,ract2 -nodevips ract1/ract1vip/255.255.255.0/eth1,ract2/ract2vip/255.255.255.0/eth1
--> For recreating nodeaapps none of these commands are needed
Syntax for clscfg and cluutil utilities
[root@ract1 install]# clscfg -h
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
clscfg -- Oracle cluster configuration tool
This tool is typically invoked as part of the Oracle Cluster Ready
Services install process. It configures cluster topology and other
settings. Use -help for information on any of these modes.
Use one of the following modes of operation.
-install - creates a new configuration
-upgrade - upgrades an existing configuration
-downgrade - downgrades an existing configuration
-add - adds a node to the configuration
-delete - deletes a node from the configuration
-local - creates a special single-node configuration for ASM
-concepts - brief listing of terminology used in the other modes
-trace - may be used in conjunction with any mode above for tracing
WARNING: Using this tool may corrupt your cluster configuration. Do not
use unless you positively know what you are doing.
[root@ract1 install]# clscfg -add -h
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
clscfg -add -nn nameA,numA,nameB,numB,... -pn privA,numA,privB,numB,...
[-hn hostA,numA,hostB,numB,...] [-t p1,p2,p3,p4]
-nn specifies nodenames in the same fashion as -nn in -install mode
-pn specifies private interconnect names as -pn in -install mode
-hn specifies hostnames in the same fashion as -hn in -install mode
-t specifies port numbers to be used by CRS daemons on the new node(s)
default ports: 49895,49896,49897,49898
WARNING: Using this tool may corrupt your cluster configuration. Do not
use unless you positively know what you are doing.
[root@ract1 install]# /u01/app/oracle/product/crs/bin/cluutil -h
usage: java ClusterUtil -sourcefile <source file location>
-destFile <destination file location> -nodelist node1[,node2,..]
Reference