UDEV setup in a Multipath env for RAC/ASM

Test Enviromment

  • OEL 6.4 with 11.2.0.4.0 RAC system

Map /dev/dm-? devices to the asm disk discovery string like /dev/asm

Display ISCSI partitions/Disks available for ASM
# dmsetup ls --tree -o device
vg_oel64-lv_swap (252:1)
 +- (8:3)
 +- (8:2)
vg_oel64-lv_root (252:0)
 +- (8:2)
grac41_disk1p1 (252:6)
 +-grac41_disk1 (252:2)
    +- (8:240)
    +- (65:32)
grac41_disk2p1 (252:7)
 +-grac41_disk2 (252:3)
    +- (65:0)
    +- (65:48)
grac41_disk3p1 (252:5)
 +-grac41_disk3 (252:4)
    +- (65:16)
    +- (65:64)

What is  Major number 252 ?
#   cat /proc/devices |  egrep 'devices|252'
..
Block devices:
252 device-mapper
-->  Driver for Device-Mapper

Availabe partitions: 
# dmsetup ls | grep p1
grac41_disk1p1    (252:5)
grac41_disk2p1    (252:7)
grac41_disk3p1    (252:6)

Map /dev/dm-? devices to /dev/asmdisk_OF-disk1 ( for example /dev/dm-5 to asmdisk_OF-disk1)
Create/Add to  12-dm-permissions.rules in /etc/udev/rules.d following entries:
ENV{DM_NAME}=="grac41_disk1p1",NAME="asmdisk_OF-disk1", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="grac41_disk2p1",NAME="asmdisk_OF-disk2", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="grac41_disk3p1",NAME="asmdisk_OF-disk3", OWNER:="grid", GROUP:="asmadmin", MODE:="660"

Change the protections for underlying sdX disks used by Device Mapper

  • Note : Don’t use Name in the 99-oracle-asmdevices.rules file as we get 2 disks for a single WWID.
  • Here we change group, owner and protection of the /dev/sdX devices created by ISCSI
Check WWID of ISCSI disks
#  multipath -ll |      grep dm-
grac41_disk3 (14f504e46494c45525237556f4c352d574b4e6c2d446e736b) dm-4 OPNFILER,VIRTUAL-DISK
grac41_disk2 (14f504e46494c4552786431396d6c2d6d4950492d52517178) dm-2 OPNFILER,VIRTUAL-DISK
grac41_disk1 (14f504e46494c45523264556151442d5261336d2d566e4250) dm-3 OPNFILER,VIRTUAL-DISK

Create/Add to  99-oracle-asmdevices.rules in /etc/udev/rules.d following entries
KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="14f504e46494c45525237556f4c352d574b4e6c2d446e736b", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="14f504e46494c4552786431396d6c2d6d4950492d52517178", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="14f504e46494c45523264556151442d5261336d2d566e4250", OWNER="grid", GROUP="asmadmin", MODE="0660"

Reload UDEV ( OEL 6 stile )
# udevadm control --reload-rules
# start_udev

Verify ASM disk protection 
# ls -l /dev/asmdisk_OF*
brw-rw----. 1 grid asmadmin 252, 5 Feb 28 10:06 /dev/asmdisk_OF-disk1
brw-rw----. 1 grid asmadmin 252, 7 Feb 28 10:06 /dev/asmdisk_OF-disk2
brw-rw----. 1 grid asmadmin 252, 6 Feb 28 10:06 /dev/asmdisk_OF-disk3

Find all attached sd disks used by our multipath configuration 
# multipath -ll | grep sd
  |- 21:0:0:0 sdt 65:48 active ready running
  `- 22:0:0:0 sdu 65:64 active ready running
  |- 17:0:0:0 sdp 8:240 active ready running
  `- 18:0:0:0 sdr 65:16 active ready running
  |- 20:0:0:0 sdq 65:0  active ready running
  `- 19:0:0:0 sds 65:32 active ready running

Verify SD disk protection
# ls -l /dev/sdt /dev/sdu /dev/sdp /dev/sdr /dev/sdq /dev/sds
brw-rw----. 1 grid asmadmin  8, 240 Feb 28 10:55 /dev/sdp
brw-rw----. 1 grid asmadmin 65,   0 Feb 28 10:55 /dev/sdq
brw-rw----. 1 grid asmadmin 65,  16 Feb 28 10:55 /dev/sdr
brw-rw----. 1 grid asmadmin 65,  32 Feb 28 10:55 /dev/sds
brw-rw----. 1 grid asmadmin 65,  48 Feb 28 10:55 /dev/sdt
brw-rw----. 1 grid asmadmin 65,  64 Feb 28 10:55 /dev/sdu

- The major number changes as there are eight major numbers reserved  ( or even more ) for SCSI devices  
- Those major numbers are 8, 65, 66, 67, 68, 79, 70, and 71

Verify your ASM disks on all nodes before you use asmca to create an OPENFILER diskgroup 

Reference:

Leave a Reply

Your email address will not be published. Required fields are marked *