Recovering OCR DG when all disks are lost

OCR diskgroup content

  • Storage for ASM SPFile
  • Storage for Voting Disk
  • Storage for OCR repository

Destroy all ASM disk headers from our OCR diskgroup

[root@grac41 Desktop]# dd if=/dev/zero  of=/dev/asm_ocr_11204_2G_disk3 bs=1024 count=1024
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.00483026 s, 217 MB/s
[root@grac41 Desktop]# dd if=/dev/zero  of=/dev/asm_ocr_11204_2G_disk1 bs=1024 count=1024
[root@grac41 Desktop]# dd if=/dev/zero  of=/dev/asm_ocr_11204_2G_disk3 bs=1024 count=1024

CW status after restart
[grid@grac41 ~]$ crsi
NAME                           TARGET     STATE           SERVER       STATE_DETAILS   
-------------------------      ---------- ----------      ------------ ------------------
ora.asm                        ONLINE     OFFLINE                        
ora.cluster_interconnect.haip  ONLINE     OFFLINE                        
ora.crf                        ONLINE     ONLINE          grac41         
ora.crsd                       ONLINE     OFFLINE                        
ora.cssd                       ONLINE     OFFLINE         STARTING       
ora.cssdmonitor                ONLINE     ONLINE          grac41         
ora.ctssd                      ONLINE     OFFLINE                        
ora.diskmon                    OFFLINE    OFFLINE                        
ora.drivers.acfs               ONLINE     OFFLINE                        
ora.evmd                       ONLINE     OFFLINE                        
ora.gipcd                      ONLINE     ONLINE          grac41         
ora.gpnpd                      ONLINE     ONLINE          grac41         
ora.mdnsd                      ONLINE     ONLINE          grac41       
--> Cssd doesn't start as none of our voting disk are accessible anymore !

CW alert log reports :
[cssd(22327)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; 
Details at (:CSSNM00070:) in /u01/app/11204/grid/log/grac41/cssd/ocssd.log

Stop CRS and start crs in exclusiv mode:
# crsctl stop crs [-f] 

Verify that all CW processes were stopped
[root@grac41 Desktop]# ps -elf | grep d.bin
0 S root      6934 28649  0  80   0 - 25826 pipe_w 11:37 pts/5    00:00:00 grep d.bin
--> Note you may need to kill remaining CW processes

Start CW stack in exclusive mode without CRS 
A new option '-nocrs' has been introduced with, which prevents the start of the ora.crsd resource.
[root@grac41 Desktop]#  $GRID_HOME/bin/crsctl start crs -excl -nocrs

Connect to ASM instance an check ASM disk status
[grid@grac41 ~]$  sqlplus / as sysasm
SQL> select dg_name,  dg.state dg_state,  dg.type, d.DISK_NUMBER dsk_no, d.MOUNT_STATUS, d.HEADER_STATUS, d.MODE_STATUS,
        d.STATE, d. PATH, d.FAILGROUP  FROM V$ASM_DISK d,  v$asm_diskgroup dg
     where dg.group_number(+)=d.group_number order by dg_name, dsk_no;
---------- ---------- ------ ------- ------- ------------ ------- -------- ------------------------------ --------------
                   1 CLOSED  CANDIDATE      ONLINE  NORMAL   /dev/asm_ocr_11204_2G_disk2
                   2 CLOSED  CANDIDATE      ONLINE  NORMAL   /dev/asm_ocr_11204_2G_disk1
                   3 CLOSED  CANDIDATE      ONLINE  NORMAL   /dev/asm_ocr_11204_2G_disk3
--> As we have erased the ASM disk header with d the header status should be shown as CANDIDATE status


Recreate diskgroup , restore backup, configure ASM SPFile and Voting disks

  2    FAILGROUP OCR_000 DISK '/dev/asm_ocr_11204_2G_disk1' NAME disk1
  3    FAILGROUP OCR_001 DISK '/dev/asm_ocr_11204_2G_disk2' NAME disk2
  4    FAILGROUP OCR_002 DISK '/dev/asm_ocr_11204_2G_disk3' NAME disk3
  5    ATTRIBUTE 'au_size'='4M',
  6       'compatible.asm' = '11.2',
  7       'compatible.rdbms' = '11.2';
Diskgroup created

Verify OCR backup ( check on all nodes to get the most current backup ! )
[grid@grac41 ~]$ ocrconfig -showbackup auto
PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy
grac41     2014/07/01 04:48:16     /u01/app/11204/grid/cdata/grac4/backup00.ocr
grac41     2014/07/01 00:48:12     /u01/app/11204/grid/cdata/grac4/backup01.ocr 

Restore the latest OCR backup
AS the CRS disk group is created & mounted the OCR can be restored - must be done as the root user:
[root@grac41 Desktop]#  $GRID_HOME/bin/ocrconfig -restore /u01/app/11204/grid/cdata/grac4/backup00.ocr

---------- ---------- ------ ------- ------- ------------ ------- -------- ------------------------------ ---------------
OCR       MOUNTED    NORMAL       0 CACHED  MEMBER      ONLINE  NORMAL   /dev/asm_ocr_11204_2G_disk1      OCR_000
OCR       MOUNTED    NORMAL       1 CACHED  MEMBER      ONLINE  NORMAL   /dev/asm_ocr_11204_2G_disk2      OCR_001
OCR       MOUNTED    NORMAL       2 CACHED  MEMBER      ONLINE  NORMAL   /dev/asm_ocr_11204_2G_disk3      OCR_002
--> Now Disk header status has changed to MEMBER 

Add voting disks
[root@grac41 ~]#  $GRID_HOME/bin/crsctl replace votedisk +OCR
CRS-4602: Failed 27 to add voting file 112b412681a04fa1bfd99c0ed4dc991c.
CRS-4602: Failed 27 to add voting file 0fd25dd59d954f80bfbd4e9f0431fb11.
CRS-4602: Failed 27 to add voting file 36bf8226cc364faebf9b616999771bbc.
Failed to replace voting disk group with +OCR.

--> Need to set asm_diskstring 
SQL> alter system set asm_diskstring="/dev/asm*";
System altered.
SQL>  show parameter asm;
NAME                     TYPE     VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups                 string     OCR
asm_diskstring                 string     /dev/asm*

[root@grac41 ~]#  $GRID_HOME/bin/crsctl replace votedisk +OCR
Successful addition of voting disk 7f78f5e770534f74bf4a00e542579bd1.
Successful addition of voting disk 7f75b3f24c064f43bfae9ca2d87329f1.
Successful addition of voting disk 8b2d9f3224c64fe1bf395647615ba722.
Successfully replaced voting disk group with +OCR.
CRS-4266: Voting file(s) successfully replaced

Recreate ASM SPFIE
SQL> create spfile='+OCR/grac4/asmparameterfile/spfileASM.ora'  from pfile='/home/grid/ASM_SPFILE/initasm.ora'   ;
File created.

Verify SPfile location, voting disks and OCR DG status after restore

--> Restart CRS 

Verify ASM SPFile location
[grid@grac41 grac41]$ asmcmd spget

[grid@grac41 grac41]$  crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   7f78f5e770534f74bf4a00e542579bd1 (/dev/asm_ocr_11204_2G_disk1) [OCR]
 2. ONLINE   7f75b3f24c064f43bfae9ca2d87329f1 (/dev/asm_ocr_11204_2G_disk2) [OCR]
 3. ONLINE   8b2d9f3224c64fe1bf395647615ba722 (/dev/asm_ocr_11204_2G_disk3) [OCR]
Located 3 voting disk(s).

Verify OCR DG 
[grid@grac41 grac41]$ asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512   4096  4194304      6132     5236             2044            1596              0             Y  OCR/

2 thoughts on “Recovering OCR DG when all disks are lost”

Leave a Reply

Your email address will not be published. Required fields are marked *