Cleanup / Removing a crashed node from OCR – RAC 10.2.0.1

Cleanup OCR

Backup your OCR
[root@ract1 ~]#   $ORA_CRS_HOME/bin/ocrconfig -export ocr_before_node_removal.exp

Verify and drop the failing instance ract3
--> Note ract3 is not available anymore 
Check current CRS status 
Name           Type           Target    State     Host
------------------------------------------------------------
ora....T3.inst application    ONLINE    OFFLINE               
ora....SM3.asm application    ONLINE    OFFLINE               
ora....T3.lsnr application    ONLINE    OFFLINE               
ora.ract3.gsd  application    ONLINE    OFFLINE               
ora.ract3.ons  application    ONLINE    OFFLINE               
ora.ract3.vip  application    ONLINE    ONLINE    ract1  
--> After crash only ora.ract3.vip  failed over to ract1 - other resources are OFFLINE

Stop all resources running on ract3
[root@ract1 ~]#  crs_stat -t | egrep 'Name|T3|t3|M3'
Name           Type           Target    State     Host        
ora....T3.inst application    ONLINE   OFFLINE               
ora....SM3.asm application    ONLINE   OFFLINE               
ora....T3.lsnr application    ONLINE   OFFLINE               
ora.ract3.gsd  application    ONLINE   OFFLINE               
ora.ract3.ons  application    ONLINE   OFFLINE               
ora.ract3.vip  application    ONLINE   ONLINE    ract1   
[root@ract1 ~]# crs_stop ora.RACT.RACT3.inst
Target set to OFFLINE for `ora.RACT.RACT3.inst`
[root@ract1 ~]# crs_stop ora.ract3.ASM3.asm
Target set to OFFLINE for `ora.ract3.ASM3.asm
[root@ract1 ~]#  crs_stop ora.ract3.LISTENER_RACT3.lsnr
Target set to OFFLINE for `ora.ract3.LISTENER_RACT3.lsnr`

[root@ract1 ~]#  crs_stop ora.ract3.gsd
Target set to OFFLINE for `ora.ract3.gsd`

[root@ract1 ~]# crs_stop ora.ract3.ons
Target set to OFFLINE for `ora.ract3.ons`

[root@ract1 ~]# crs_stop ora.ract3.vip
Attempting to stop `ora.ract3.vip` on member `ract1`
Stop of `ora.ract3.vip` on member `ract1` succeeded.

[root@ract1 ~]# crs_stat -t | egrep 'Name|T3|t3|M3'
Name           Type           Target    State     Host        
ora....T3.inst application    OFFLINE   OFFLINE               
ora....SM3.asm application    OFFLINE   OFFLINE               
ora....T3.lsnr application    OFFLINE   OFFLINE               
ora.ract3.gsd  application    OFFLINE   OFFLINE               
ora.ract3.ons  application    OFFLINE   OFFLINE               
ora.ract3.vip  application    OFFLINE   OFFLINE      

Remove RDBMS instance
[oracle@ract1 ~]$  srvctl remove instance -d ract -i RACT3
Remove instance RACT3 from the database ract? (y/[n]) y

Remove ASM instance 
$ srvctl remove asm -f -n ract3

Remove RAC listener
The only way to remove the listener resources is to use the command 'crs_unregister', 
please use this command only in this particular scenario:
[root@ract1 ~]# crs_stat | grep lsnr
NAME=ora.ract1.LISTENER_RACT1.lsnr
NAME=ora.ract2.LISTENER_RACT2.lsnr
NAME=ora.ract3.LISTENER_RACT3.lsnr
[root@ract1 ~]# crs_unregister ora.ract3.LISTENER_RACT3.lsnr

Remove remaining nodeapps 
[oracle@ract1 ~]$  crs_stat -t | egrep 'Name|T3|t3|M3'
Name           Type           Target    State     Host        
ora.ract3.gsd  application    OFFLINE   OFFLINE               
ora.ract3.ons  application    OFFLINE   OFFLINE               
ora.ract3.vip  application    OFFLINE   OFFLINE 

[root@ract1 ~]# srvctl  remove  nodeapps -n ract3 
Please confirm that you intend to remove the node-level applications on node ract3 (y/[n]) y

Verify that all resource linked to ract3 are removed
[root@ract1 ~]#   crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....T1.inst application    ONLINE    ONLINE    ract1       
ora....T2.inst application    ONLINE    ONLINE    ract2       
ora.RACT.db    application    ONLINE    ONLINE    ract2       
ora....SM1.asm application    ONLINE    ONLINE    ract1       
ora....T1.lsnr application    ONLINE    ONLINE    ract1       
ora.ract1.gsd  application    ONLINE    ONLINE    ract1       
ora.ract1.ons  application    ONLINE    ONLINE    ract1       
ora.ract1.vip  application    ONLINE    ONLINE    ract1       
ora....SM2.asm application    ONLINE    ONLINE    ract2       
ora....T2.lsnr application    ONLINE    ONLINE    ract2       
ora.ract2.gsd  application    ONLINE    ONLINE    ract2       
ora.ract2.ons  application    ONLINE    ONLINE    ract2       
ora.ract2.vip  application    ONLINE    ONLINE    ract2 


Remove node by running ./rootdeletenode.sh
[root@ract1 ~]# olsnodes -n
ract1   1
ract2   2
ract3   3

[root@ract1 ~]# cd $ORA_CRS_HOME/install
[root@ract1 install]#  ./rootdeletenode.sh ract3,3
CRS-0210: Could not find resource 'ora.ract3.LISTENER_RACT3.lsnr'.
CRS-0210: Could not find resource 'ora.ract3.ons'.
CRS-0210: Could not find resource 'ora.ract3.vip'.
CRS-0210: Could not find resource 'ora.ract3.gsd'.
CRS-0210: Could not find resource ora.ract3.vip.
CRS nodeapps are deleted successfully
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully deleted 14 values from OCR.
Key SYSTEM.css.interfaces.noderact3 marked for deletion is not there. Ignoring.
Successfully deleted 5 keys from OCR.
Node deletion operation successful.
'ract3,3' deleted successfully

[root@ract1 install]#  olsnodes -n
ract1   1
ract2   2      

Cleanup CRS and Rdbms Inventory   
$ cd $ORA_CRS_HOME/oui/bin
$ ./runInstaller
  Installed products
   OraCrs10g_home
    Cluster Nodes
     ract1
     ract2
     ract3
   OraDb10g_home1
    Cluster Nodes
     ract1
     ract2
     ract3
[oracle@ract1 bin]$  cd $ORA_CRS_HOME/oui/bin
[oracle@ract1 bin]$  ./runInstaller -updateNodeList ORACLE_HOME=$ORA_CRS_HOME "CLUSTER_NODES={ract1,ract2}" CRS=true
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
'UpdateNodeList' was successful.

[oracle@ract1 bin]$  cd $ORACLE_HOME/oui/bin
[oracle@ract1 bin]$  $ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME  "CLUSTER_NODES={ract1,ract2}"
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
'UpdateNodeList' was successful.

Verify Cluster Nodes
$ ./runInstaller
  Installed products
   OraCrs10g_home
    Cluster Nodes
     ract1
     ract2
   OraDb10g_home1
    Cluster Nodes
     ract1
     ract2

Cleanup crashed Node ( ract3 ) after reboot

Be carefull when deleting files - depending on your setup you may damage a 2.nd database install
# To remove RDBMS and CRS software run ( both crs and rdbms are installed under  /u01/app/oracle/ )
rm -rf  /u01/app/oracle/*   <-- dangerous !!
#
rm /etc/oracle/*
rm -f /etc/init.d/init.cssd
rm -f /etc/init.d/init.crs
rm -f /etc/init.d/init.crsd
rm -f /etc/init.d/init.evmd
rm -f /etc/rc2.d/K96init.crs
rm -f /etc/rc2.d/S96init.crs
rm -f /etc/rc3.d/K96init.crs
rm -f /etc/rc3.d/S96init.crs
rm -f /etc/rc5.d/K96init.crs
rm -f /etc/rc5.d/S96init.crs
rm -Rf /etc/oracle/scls_scr
rm -f /etc/inittab.crs
cp /etc/inittab.orig /etc/inittab

--> reboot cleanup node again and check that no cluster process has survived the cleanup !

Verify that no CRS process survived the cleanup
# ps -elf |grep d.bin

Error: srvctl remove instance fails IF INSTANCE: TARGET=ONLINE AND STATE=OFFLINE

[root@ract1 ~]# srvctl remove instance -d ract -i RACT3
root@ract1 ~]#  crs_stat  -t | egrep 'Name|t3|T3'
Name           Type           Target    State     Host        
ora....T3.inst application    ONLINE    OFFLINE       
--> Target for Instance RACT3 still ONLINE !!
See  BUG  4423294 : DELETE INSTANCE THROUGH SRVM FAILS IF INSTANCE: 
                   TARGET=ONLINE AND STATE=OFFLINE crs_stop  should fix !

[root@ract1 ~]# crs_stop ora.RACT.RACT3.inst
Target set to OFFLINE for `ora.RACT.RACT3.inst`
[root@ract1 ~]#  crs_stat -t | egrep 'Name|t3|T3'
Name           Type           Target    State     Host        
ora....T3.inst application    OFFLINE   OFFLINE     

Now srvctl remove instance should work
[root@ract1 ~]# srvctl remove instance -d ract -i RACT3

 

Reference

2 thoughts on “Cleanup / Removing a crashed node from OCR – RAC 10.2.0.1”

  1. I’m extremely impressed along with your writing abilities and also with the
    format for your blog. Is that this a paid theme or did
    you modify it yourself? Either way stay up the nice high quality writing, it is uncommon to
    peer a nice blog like this one these days..

    For an incredible solution please click the link to this page
    :: http://cnn.com (Tabitha)

    1. Hi Tabitha,

      thx for your nice reply.
      I have used and modified default wordpress theme : Twenty Twelve

      br
      Helmut

Leave a Reply to Tabitha Cancel reply

Your email address will not be published. Required fields are marked *