OCR disk group content
- tested with GRID 11.2.0.4
- Storage for ASM SPFile
- Storage for Voting Disk
- Storage for OCR repository
Setup test case
Destroy ASM disk header from our OCR diskgroup [root@grac41 Desktop]# dd if=/dev/zero of=/dev/asm_ocr_11204_2G_disk3 bs=1024 count=1024 1024+0 records in 1024+0 records out 1048576 bytes (1.0 MB) copied, 0.00483026 s, 217 MB/s Verify CW status after startup [root@grac41 Desktop]# crsctl start crs CRS-4123: Oracle High Availability Services has been started. Monitor CRS startup [grid@grac41 ~]$ watch crsi NAME TARGET STATE SERVER STATE_DETAILS ------------------------- ---------- ---------- ------------ ------------------ ora.asm ONLINE INTERMEDIATE grac41 OCR not started ora.cluster_interconnect.haip ONLINE ONLINE grac41 ora.crf ONLINE ONLINE grac41 ora.crsd ONLINE OFFLINE ora.cssd ONLINE ONLINE grac41 ora.cssdmonitor ONLINE ONLINE grac41 ora.ctssd ONLINE ONLINE grac41 OBSERVER ora.diskmon OFFLINE OFFLINE ora.drivers.acfs ONLINE OFFLINE ora.evmd ONLINE INTERMEDIATE grac41 ora.gipcd ONLINE ONLINE grac41 ora.gpnpd ONLINE ONLINE grac41 ora.mdnsd ONLINE ONLINE grac41 RAC Alertlog [/u01/app/11204/grid/bin/oraagent.bin(15374)]CRS-5019:All OCR locations are on ASM disk groups [OCR], and none of these disksgroups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11204/grid/log/grac41/agent/ohasd/oraagent_grid/oraagent_grid.log". Agent Log 2014-07-05 09:23:32.627: [ora.asm][1002428160]{0:0:2} [start] ORA-15032: not all alterations performed ORA-15040: diskgroup is incomplete ORA-15042: ASM disk "0" is missing from group number "1" Check trace for errors: [grid@grac41 log]$ fn.sh ORA- | egrep 'TraceFile|2014-07-05 09:' TraceFileName: ./grac41/agent/ohasd/oraagent_grid/oraagent_grid.l01 2014-07-05 09:23:32.627: [ora.asm][1002428160]{0:0:2} [start] ORA-15032: not all alterations performed 2014-07-05 09:23:32.640: [ora.asm][1002428160]{0:0:2} [start] ORA-15100: invalid or missing diskgroup name Check voting disks [grid@grac41 log]$ crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE a4452602e3be4f42bf5467f41d96e46a (/dev/asm_ocr_11204_2G_disk1) [OCR] 2. ONLINE 293e6baa9cbc4f90bf01f52b1b9019fa (/dev/asm_ocr_11204_2G_disk2) [OCR] 3. OFFLINE 69e2423e2cff4f64bf16c1346a900803 () [] Located 3 voting disk(s). --> Voting Disk 3 is OFFLINE t@grac41 ~]# ocrcheck -local Status of Oracle Local Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 2676 Available space (kbytes) : 259444 ID : 1855884304 Device/File Name : /u01/app/11204/grid/cdata/grac41.olr Device/File integrity check succeeded Local registry integrity check succeeded Logical corruption check succeeded --> OLR ok [root@grac41 ~]# ocrcheck PROT-602: Failed to retrieve data from the cluster registry PROC-26: Error while accessing the physical storage [root@grac41 ~]# ocrcheck -config Oracle Cluster Registry configuration is : Device/File Name : +OCR --> Local OLR ok - Cluster OCR not ONLINE due to missing voting disk Find the missing disk name of our 3.rd ASM disk SQL> select dg.name dg_name, dg.state dg_state, dg.type, d.DISK_NUMBER dsk_no, d.MOUNT_STATUS, d.HEADER_STATUS, d.MODE_STATUS, 2 d.STATE, d. PATH, d.FAILGROUP FROM V$ASM_DISK d, v$asm_diskgroup dg 3 where dg.group_number(+)=d.group_number order by dg_name, dsk_no; DG_NAME DG_STATE TYPE DSK_NO MOUNT_S HEADER_STATU MODE_ST STATE PATH FAILGROUP ---------- ---------- ------ ------- ------- ------------ ------- -------- ------------------------------ --------------- OCR DISMOUNTED 1 CLOSED MEMBER ONLINE NORMAL /dev/asm_ocr_11204_2G_disk2 OCR DISMOUNTED 2 CLOSED MEMBER ONLINE NORMAL /dev/asm_ocr_11204_2G_disk1 OCR DISMOUNTED 3 CLOSED CANDIDATE ONLINE NORMAL /dev/asm_ocr_11204_2G_disk3 Check the ASM disk header this with kfed [root@grac41 Desktop]# kfed read /dev/asm_ocr_11204_2G_disk3 kfbh.endian: 0 ; 0x000: 0x00 kfbh.hard: 0 ; 0x001: 0x00 kfbh.type: 0 ; 0x002: KFBTYP_INVALID kfbh.datfmt: 0 ; 0x003: 0x00 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 0 ; 0x008: file=0 kfbh.check: 0 ; 0x00c: 0x00000000 kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 7F4C5C91B400 00000000 00000000 00000000 00000000 [................] Repeat 255 times KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0] --> ASM disk header erased Verify the disk header from the remaining disk [root@grac41 Desktop]# kfed read /dev/asm_ocr_11204_2G_disk1 | egrep 'type|name' kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD kfdhdb.dskname: OCR_0003 ; 0x028: length=8 kfdhdb.grpname: OCR ; 0x048: length=3 kfdhdb.fgname: OCR_0003 ; 0x068: length=8 kfdhdb.capname: ; 0x088: length=0 [root@grac41 Desktop]# kfed read /dev/asm_ocr_11204_2G_disk2 | egrep 'type|name' kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD kfdhdb.dskname: OCR_0000 ; 0x028: length=8 kfdhdb.grpname: OCR ; 0x048: length=3 kfdhdb.fgname: OCR_0000 ; 0x068: length=8 kfdhdb.capname: ; 0x088: length=0 --> Disks asm_ocr_11204_2G_disk1 and asm_ocr_11204_2G_disk2 are ok !
Add the failed disk back again to OCR DG
Mount OCR DG with force option and re-add the repaired disk SQL> alter diskgroup OCR mount force; Diskgroup altered. SQL> @dg DG_NAME DG_STATE TYPE DSK_NO MOUNT_S HEADER_STATU MODE_ST STATE PATH FAILGROUP ---------- ---------- ------ ------- ------- ------------ ------- -------- ------------------------------ --------------- OCR MOUNTED NORMAL 0 CACHED MEMBER ONLINE NORMAL /dev/asm_ocr_11204_2G_disk2 OCR_0000 OCR MOUNTED NORMAL 1 MISSING UNKNOWN OFFLINE NORMAL OCR_0001 OCR MOUNTED NORMAL 3 CACHED MEMBER ONLINE NORMAL /dev/asm_ocr_11204_2G_disk1 OCR_0003 3 CLOSED CANDIDATE ONLINE NORMAL /dev/asm_ocr_11204_2G_disk3 SQL> alter diskgroup OCR add disk '/dev/asm_ocr_11204_2G_disk3'; Diskgroup altered. SQL> @dg DG_NAME DG_STATE TYPE DSK_NO MOUNT_S HEADER_STATU MODE_ST STATE PATH FAILGROUP ---------- ---------- ------ ------- ------- ------------ ------- -------- ------------------------------ --------------- OCR MOUNTED NORMAL 0 CACHED MEMBER ONLINE NORMAL /dev/asm_ocr_11204_2G_disk2 OCR_0000 OCR MOUNTED NORMAL 1 MISSING UNKNOWN OFFLINE NORMAL OCR_0001 OCR MOUNTED NORMAL 2 CACHED MEMBER ONLINE NORMAL /dev/asm_ocr_11204_2G_disk3 OCR_0002 OCR MOUNTED NORMAL 3 CACHED MEMBER ONLINE NORMAL /dev/asm_ocr_11204_2G_disk1 OCR_0003 Testing mount/remount operation before CW restart SQL> alter diskgroup OCR dismount force; Diskgroup altered. SQL> alter diskgroup OCR mount; Diskgroup altered. SQL> @dg DG_NAME DG_STATE TYPE DSK_NO MOUNT_S HEADER_STATU MODE_ST STATE PATH FAILGROUP ---------- ---------- ------ ------- ------- ------------ ------- -------- ------------------------------ --------------- OCR MOUNTED NORMAL 0 CACHED MEMBER ONLINE NORMAL /dev/asm_ocr_11204_2G_disk2 OCR_0000 OCR MOUNTED NORMAL 1 MISSING UNKNOWN OFFLINE FORCING OCR_0001 OCR MOUNTED NORMAL 2 CACHED MEMBER ONLINE NORMAL /dev/asm_ocr_11204_2G_disk3 OCR_0002 OCR MOUNTED NORMAL 3 CACHED MEMBER ONLINE NORMAL /dev/asm_ocr_11204_2G_disk1 OCR_0003 Restart CW and verify voting disk status - Note disk with FORCING state will be cleaned up - Missing Vote disk will be added automatically DG_NAME DG_STATE TYPE DSK_NO MOUNT_S HEADER_STATU MODE_ST STATE PATH FAILGROUP ---------- ---------- ------ ------- ------- ------------ ------- -------- ------------------------------ --------------- OCR MOUNTED NORMAL 0 CACHED MEMBER ONLINE NORMAL /dev/asm_ocr_11204_2G_disk2 OCR_0000 OCR MOUNTED NORMAL 2 CACHED MEMBER ONLINE NORMAL /dev/asm_ocr_11204_2G_disk3 OCR_0002 OCR MOUNTED NORMAL 3 CACHED MEMBER ONLINE NORMAL /dev/asm_ocr_11204_2G_disk1 OCR_0003 [root@grac41 Desktop]# crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE a4452602e3be4f42bf5467f41d96e46a (/dev/asm_ocr_11204_2G_disk1) [OCR] 2. ONLINE 293e6baa9cbc4f90bf01f52b1b9019fa (/dev/asm_ocr_11204_2G_disk2) [OCR] 3. ONLINE a0967baeb88b4f45bf3ae5da678fecc4 (/dev/asm_ocr_11204_2G_disk3) [OCR]