Relocate OCR and Voting Disks to a different ASM diskgroup ( 11.2.0.4 )

Create a new diskgroup for OCR and Voting disk

Use ascma and create new Data group named OCR and verify that this datagroup is mounted
$ asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512   4096  1048576     40944    25698            10236            7731              0             Y  DATA/
MOUNTED  NORMAL  N         512   4096  1048576      6141     5730             2047            1841              0             N  OCR/
Attention:
  To avoid error CRS-4602: Failed 27 to add voting file .. during running $GRID_HOME/bin/crsctl  replace votedisk  double check that the
  the newly created diskgroup is mounted on any cluster instances by running 
$ asmcmd lsdg
$ asmcmd lsdsk 
on each instance.

 

Relocate OCR from +DATA to +OCR diskgroup

Check the current OCR location
# $GRID_HOME/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
     Version                  :          3
     Total space (kbytes)     :     262120
     Used space (kbytes)      :       3664
     Available space (kbytes) :     258456
     ID                       :  630679368
     Device/File Name         :      +DATA
                                    Device/File integrity check succeeded

Add +OCR datagroup as new OCR location and verify status
#  $GRID_HOME/bin/ocrconfig -add +OCR
#  $GRID_HOME/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
     Version                  :          3
     Total space (kbytes)     :     262120
     Used space (kbytes)      :       3664
     Available space (kbytes) :     258456
     ID                       :  630679368
     Device/File Name         :      +DATA
                                    Device/File integrity check succeeded
     Device/File Name         :       +OCR
                                    Device/File integrity check succeeded 

Delete the old OCR location 
# $GRID_HOME/bin/ocrconfig -delete +DATA
# $GRID_HOME/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
     Version                  :          3
     Total space (kbytes)     :     262120
     Used space (kbytes)      :       3664
     Available space (kbytes) :     258456
     ID                       :  630679368
     Device/File Name         :       +OCR
                                    Device/File integrity check succeeded

 

Relocate voting disks to a new ASM diskgroup

Check current status :
# $GRID_HOME/bin/crsctl   query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   7c7a1eb193644fc3bffd655ae8e43462 (/dev/asmdisk1_udev_sdb1) [DATA]
 2. ONLINE   6590ede5237e4f37bfe6aa3d196fe972 (/dev/asmdisk2_udev_sdc1) [DATA]
 3. ONLINE   3a7c7b334faf4fdcbf05a246eddf5404 (/dev/asmdisk3_udev_sdd1) [DATA]
Located 3 voting disk(s).

Relocate Voting Disks
$  $GRID_HOME/bin/crsctl  replace votedisk +OCR
Successful addition of voting disk 0850980df1024fcfbfc077b9c6935a17.
Successful addition of voting disk 0ca6719843734f16bfb62f22f94f399c.
Successful addition of voting disk 4383a5ad43324fa2bff759ce20ba0c3f.
Successful deletion of voting disk 10c81d1ce5a14fb6bf35cbb22fff3ebf.:1
Successful deletion of voting disk 98010612be6b4fc9bf3bc1b186d8758d.
Successful deletion of voting disk 9688bec3914d4f70bfc959664ddd8584.
Successfully replaced voting disk group with +OCR.
CRS-4266: Voting file(s) successfully replaced

#  $GRID_HOME/bin/crsctl   query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   b0e94e5d83054fe9bf58b6b98bfacd65 (/dev/asmdisk5_udev_sdf1) [OCR]
 2. ONLINE   88c2a08b4c8c4f85bf0109e0990388e4 (/dev/asmdisk6_udev_sdg1) [OCR]
 3. ONLINE   1108f9a41e814fb2bfed879ff0039dd0 (/dev/asmdisk7_udev_sdh1) [OCR]

 

Error : CRS-4602: Failed 27 to add voting file 9e50b0768b874fc7bf173cd55ce106a3

# $GRID_HOME/bin/crsctl  replace votedisk +OCR
CRS-4602: Failed 27 to add voting file 9e50b0768b874fc7bf173cd55ce106a3.
CRS-4602: Failed 27 to add voting file 285788c4af634f99bfdd9edbf30dacc0.
CRS-4602: Failed 27 to add voting file 55ace1807eca4f0dbf0545faa57122ac.
Failure 27 with Cluster Synchronization Services while deleting voting disk.
Failure 27 with Cluster Synchronization Services while deleting voting disk.
Failure 27 with Cluster Synchronization Services while deleting voting disk.
Failed to replace voting disk group with +OCR.
CRS-4000: Command Replace failed, or completed with errors.

Reading Traces:
alertgrac41.log
[cssd(3456)]CRS-1650:Configuration change 1381848175:0 rejected by node 2 grac42; details at (:CSSNM000066:) 
in /u01/app/11204/grid/log/grac41/cssd/ocssd.log
2013-10-15 16:42:56.068:
[cssd(3456)]CRS-1604:CSSD voting file is offline: /dev/asmdisk6_udev_sdg1; details at (:CSSNM00069:) 
in /u01/app/11204/grid/log/grac41/cssd/ocssd.log.
2013-10-15 16:42:56.079:

Note: Always check your asm_diskstring when getting CRS-4602 errors !


grac41/cssd/ocssd.log:
2013-10-15 15:00:00.191: [    CSSD][1646704384](:CSSNM00069:)clssnmvDeconfigureVFs: CSS voting file /dev/asmdisk6_udev_sdg1 is not usable.
2013-10-15 15:00:00.191: [    CSSD][1646704384]clssnmvDiskStateChange: state from pending to deconfigured disk /dev/asmdisk6_udev_sdg1
2013-10-15 15:00:00.191: [    CSSD][1646704384](:CSSNM00069:)clssnmvDeconfigureVFs: CSS voting file /dev/asmdisk7_udev_sdh1 is not usable.
2013-10-15 15:00:00.191: [    CSSD][1646704384]clssnmvDiskStateChange: state from pending to deconfigured disk /dev/asmdisk7_udev_sdh1
2013-10-15 15:00:00.191: [    CSSD][1646704384](:CSSNM00069:)clssnmvDeconfigureVFs: CSS voting file /dev/asmdisk5_udev_sdf1 is not usable.
2013-10-15 15:00:00.191: [    CSSD][1646704384]clssnmvDiskStateChange: state from pending to deconfigured disk /dev/asmdisk5_udev_sdf1
2013-10-15 15:00:00.191: [    CSSD][1646704384]clssnmCompleteGMReq: Completed request type 1 with status 6

2013-10-15 15:00:00.192: [    CSSD][1636685568]clssnmvVoteDiskValidation: Voting disk(/dev/asmdisk7_udev_sdh1) cluster GUID mismatch
2013-10-15 15:00:00.192: [    CSSD][1636685568]clssnmvWorkerThread: disk /dev/asmdisk7_udev_sdh1 corrupted
2013-10-15 15:00:00.192: [    CSSD][1636685568]clssnmvDiskAvailabilityChange: voting file /dev/asmdisk7_udev_sdh1 now offline
2013-10-15 15:00:00.192: [    CSSD][1641441024]clssnmvVoteDiskValidation: Voting disk(/dev/asmdisk6_udev_sdg1) cluster GUID mismatch
-->  Here we know that node 2 has rejected the configurations change and it seems that  /dev/asmdisk6_udev_sdg1, /dev/asmdisk7 and
    /dev/asmdisk5_udev_sdf1 are not usable

Debug steps - Go to grac42 and check ASM status
$   $GRID_HOME/bin/asmcmd lsdsk
Connected to an idle instance.   <-- No disks reported 
$  $GRID_HOME/bin/asmcmd lsdg
Connected to an idle instance.
ASMCMD-8102: no connection to Oracle ASM; command requires Oracle ASM to run
$ ls /dev/asm*
/dev/asmdisk1_udev_sdb1  /dev/asmdisk2_udev_sdc1  /dev/asmdisk3_udev_sdd1  /dev/asmdisk4_udev_sde1
--> ASM instance not up and running - because diskgroup OCR and can not moumt  disks dev/asmdisk5_udev_sdf1, ...
    because OS has not discovered these disks 

Fix: 
     Configure udev rules for disks /dev/asmdisk6_udev_sdg1, /dev/asmdisk7 and /dev/asmdisk5_udev_sdf1  on grac32

 

3 thoughts on “Relocate OCR and Voting Disks to a different ASM diskgroup ( 11.2.0.4 )

  1. stuti

    Hi, Nicely described, almost the same activity i intend to perform.
    i have to have one confirmation:
    will this all cause any downtime????

    Reply
    1. admin Post author

      No these operations are not expected to cause any downtime.
      But as as always in a RAC and high production env you should test
      and verify any change on a test cluster first.

      Reply
  2. Pingback: Move ocr and votedisk to different location | hiteshgondalia(OCE,OCP)

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>