ASM doesn’t start due to CRS-1714 error

Startup and check CRS and verify your disk groups 
# $GRID_HOME/bin/crsctl  start crs
CRS-4123: Oracle High Availability Services has been started.

# $GRID_HOME/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager

$ asmcmd lsdg
Connected to an idle instance.
ASMCMD-8102: no connection to Oracle ASM; command requires Oracle ASM to run

--> From GRID alert.log 
[cssd(10157)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; 
Details at (:CSSNM00070:) in /u01/app/11204/grid/log/grac42/cssd/ocssd.log

$  /etc/init.d/oracleasm listdisks
--> No ASM disks configured - Installation uses udev devices!

Get your disk discovery string and verify your /dev/asm directory 
$GRID_HOME/bin/gpnptool get
DiscoveryString="/dev/asm*
SPFile="+DATA/grac4/asmparameterfile/registry.253.825944079"

$  ls -l /dev/asm*
brw-rw----. 1 grid asmadmin 252, 5 Mar 18 13:42 /dev/asmdisk_OF-disk1
brw-rw----. 1 grid asmadmin 252, 6 Mar 18 13:42 /dev/asmdisk_OF-disk2
brw-rw----. 1 grid asmadmin 252, 7 Mar 18 13:42 /dev/asmdisk_OF-disk3

$ kfed read /dev/asmdisk_OF-disk1 | egrep 'dskname|grpname'
kfdhdb.dskname:       OPENFILER_DG_0000 ; 0x028: length=17
kfdhdb.grpname:            OPENFILER_DG ; 0x048: length=12
[grid@grac42 grac42]$  kfed read /dev/asmdisk_OF-disk2 | egrep 'dskname|grpname'
kfdhdb.dskname:       OPENFILER_DG_0001 ; 0x028: length=17
kfdhdb.grpname:            OPENFILER_DG ; 0x048: length=12
[grid@grac42 grac42]$ kfed read /dev/asmdisk_OF-disk3  | egrep 'dskname|grpname'
kfdhdb.dskname:       OPENFILER_DG_0002 ; 0x028: length=17
kfdhdb.grpname:            OPENFILER_DG ; 0x048: length=12

Verify  voting file location locally
# more /etc/oracle/ocr.loc
ocrconfig_loc=+OCR
local_only=false

Verify voting file location on a working instance
$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   b0e94e5d83054fe9bf58b6b98bfacd65 (/dev/asmdisk5_udev_sdf1) [OCR]
 2. ONLINE   88c2a08b4c8c4f85bf0109e0990388e4 (/dev/asmdisk6_udev_sdg1) [OCR]
 3. ONLINE   1108f9a41e814fb2bfed879ff0039dd0 (/dev/asmdisk7_udev_sdh1) [OCR]

Check ASM devices 
ON Working Node
[root@grac41 rules.d]# ls /dev/asm*
/dev/asmdisk10_ssd3      /dev/asmdisk2_test       /dev/asmdisk4_udev_sde1  /dev/asmdisk8_ssd1     /dev/asmdisk_OF-disk3
/dev/asmdisk1_fra        /dev/asmdisk2_udev_sdc1  /dev/asmdisk5_udev_sdf1  /dev/asmdisk9_ssd2
/dev/asmdisk1_test       /dev/asmdisk3_test       /dev/asmdisk6_udev_sdg1  /dev/asmdisk_OF-disk1
/dev/asmdisk1_udev_sdb1  /dev/asmdisk3_udev_sdd1  /dev/asmdisk7_udev_sdh1  /dev/asmdisk_OF-disk2

Failing Node
grid@grac42 grac42]$  ls -l /dev/asm*
brw-rw----. 1 grid asmadmin 252, 5 Mar 18 13:42 /dev/asmdisk_OF-disk1
brw-rw----. 1 grid asmadmin 252, 6 Mar 18 13:42 /dev/asmdisk_OF-disk2
brw-rw----. 1 grid asmadmin 252, 7 Mar 18 13:42 /dev/asmdisk_OF-disk3

--> All disks for  Diskgroup OCR are missing from /dev/asm* 

Solution : Fix udev file and verify that our OCR diskgroup can be mounted
$ udevadm control --reload-rules
$ start_udev

Verify ASM disks:
[root@grac43 rules.d]#  $GRID_HOME/bin/kfed read  /dev/asmdisk1_udev_sdf1| egrep 'dskname|grpname'
kfdhdb.dskname:                OCR_0000 ; 0x028: length=8
kfdhdb.grpname:                     OCR ; 0x048: length=3
[root@grac43 rules.d]# $GRID_HOME/bin/kfed read  /dev/asmdisk1_udev_sdg1| egrep 'dskname|grpname'
kfdhdb.dskname:                OCR_0001 ; 0x028: length=8
kfdhdb.grpname:                     OCR ; 0x048: length=3
[root@grac43 rules.d]#  $GRID_HOME/bin/kfed read  /dev/asmdisk1_udev_sdh1| egrep 'dskname|grpname'
kfdhdb.dskname:                OCR_0002 ; 0x028: length=8
kfdhdb.grpname:                     OCR ; 0x048: length=3

Restart CRS ( as root ) and verify ASM status
# $GRID_HOME/bin/crsctl  stop crs -f
# $GRID_HOME/bin/crsctl start crs

$ $GRID_HOME/bin/asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512   4096  1048576     40944    20431            10236            5097              0             N  DATA/
MOUNTED  EXTERN  N         512   4096  1048576    204797   173814                0          173814              0             N  FRA/
MOUNTED  NORMAL  N         512   4096  1048576      6141     5091             2047            1522              0             Y  OCR/
MOUNTED  EXTERN  N         512   4096  1048576      5739     4565                0            4565              0             N  OPENFILER_DG/
MOUNTED  NORMAL  N         512   4096  1048576      3057     1995             1019             488              0             N  SSD/

 

Mapping Partitions with UDEV:

Mapping Disks with UDEV:

2 thoughts on “ASM doesn’t start due to CRS-1714 error

  1. Pingback: How To Fix Crs Error Codes in Windows

  2. Koko

    Awesome well detailed

    Quick question what would have caused the issue with the udev rules to be corrupted or changed if they were working OK before ?

    Pls advise

    Koko

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>