CRS-210 error and OCR corruption – Error: could not find ora.ract3.vip

Problem description

srvctl remove  nodeapps returns ORA-210
Even using  srvctl remove -f doesn't fix the problem 
--> OCR repository corrupted 

[root@ract3 ~]# srvctl remove  nodeapps  -n ract3 
Please confirm that you intend to remove the node-level applications on node ract3 (y/[n]) y
CRS-0210: Could not find resource ora.ract3.vip.

Even using  srvctl remove nodeapps  -f doesn't fix the problem --> OCR repository corrupted 
[root@ract3 ~]# srvctl remove  nodeapps  -n ract3 -f 
Please confirm that you intend to remove the node-level applications on node ract3 (y/[n]) y
CRS-0210: Could not find resource ora.ract3.vip.

Status: VIP still reported with srvctl with config nodeapps but can't be removed 
[root@ract3 ~]# srvctl config nodeapps -n ract3  -a -g -s -l
VIP exists.: /ract3vip.example.com/192.168.1.137/255.255.255.0/eth1
GSD exists.
ONS daemon exists.
Listener does not exist.
--> OCR repository seems to be corrupted

Repairing or Restoring an Inconsistent OCR in RAC by following ( Doc ID 268937.1)

 
$ORA_CRS_HOME/bin/ocrconfig -showbackup
[root@ract1 rac_db1]#  $ORA_CRS_HOME/bin/ocrconfig -showbackup
ract1     2014/04/11 14:54:52     /u01/app/oracle/product/crs/cdata/crs10g
ract1     2014/04/10 18:32:29     /u01/app/oracle/product/crs/cdata/crs10g
ract1     2014/04/10 12:55:12     /u01/app/oracle/product/crs/cdata/crs10g
ract1     2014/04/10 12:55:12     /u01/app/oracle/product/crs/cdata/crs10g
ract2     2014/04/07 08:06:05     /u01/app/oracle/product/crs/cdata/crs10g

Disable CRS on all nodes and restore OCR from ract2 backup  2014/04/07 08:06:05

root@ract2 ~]# /etc/init.d/init.crs disable
Automatic startup disabled for system boot.
root@ract2 ~]# /etc/init.d/init.crs stop
--> Run above commands on all nodes of your cluster

Locate OCR backup to be restored  
[root@ract2 ~]# cd /u01/app/oracle/product/crs/cdata/crs10g
[root@ract2 crs10g]# ls
backup00.ocr  backup01.ocr  backup02.ocr  day.ocr  week.ocr
[root@ract2 crs10g]# ls -rlt
total 20904
-rw-r--r-- 1 root root 4050944 Apr  7 08:06 week.ocr
-rw-r--r-- 1 root root 4063232 Apr  7 18:15 backup02.ocr
-rw-r--r-- 1 root root 4063232 Apr  9 14:37 day.ocr
-rw-r--r-- 1 root root 4063232 Apr  9 14:37 backup01.ocr
-rw-r--r-- 1 root root 5136384 Apr  9 18:37 backup00.ocr

Restore OCR from backup
[root@ract2 crs10g]# cd $ORA_CRS_HOME/bin
[root@ract2 bin]#   ./ocrconfig -restore  /u01/app/oracle/product/crs/cdata/crs10g/week.ocr

Re-enable CRS  on all nodes: 
[root@ract2 crs10g]#    /etc/init.d/init.crs enable
and reboot all nodes 

Verify cluster status after reboot 
[root@ract1 ~]# crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....T1.inst application    ONLINE    ONLINE    ract1       
ora....T2.inst application    ONLINE    ONLINE    ract2       
ora.RACT.db    application    ONLINE    ONLINE    ract2       
ora....SM1.asm application    ONLINE    ONLINE    ract1       
ora....T1.lsnr application    ONLINE    ONLINE    ract1       
ora.ract1.gsd  application    ONLINE    ONLINE    ract1       
ora.ract1.ons  application    ONLINE    ONLINE    ract1       
ora.ract1.vip  application    ONLINE    ONLINE    ract1       
ora....SM2.asm application    ONLINE    ONLINE    ract2       
ora....T2.lsnr application    ONLINE    ONLINE    ract2       
ora.ract2.gsd  application    ONLINE    ONLINE    ract2       
ora.ract2.ons  application    ONLINE    ONLINE    ract2

 

Reference

  • Repairing or Restoring an Inconsistent OCR in RAC (Doc ID 268937.1)

Leave a Reply

Your email address will not be published. Required fields are marked *