Fixing problem 2: Disk DATA_0001 is handled by ASMLIB at Node 1 but is using UDEV at grac42,grac43
Drop ASMLIB disk ( be carefull here you can damage your data ! ) [root@grac41 network-scripts]# /etc/init.d/oracleasm deletedisk DATA2 Removing ASM disk "DATA2": [ OK ] [root@grac41 network-scripts]# /etc/init.d/oracleasm scandisks Scanning the system for Oracle ASMLib disks: [ OK ] [root@grac41 network-scripts]# /etc/init.d/oracleasm listdisks --> repeat this on grac42,grac43 - if needed Verify and cleanup disk header: [root@grac42 ~]# kfed read /dev/asmdisk1_udev_sdc1 | egrep 'type|name' kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD kfdhdb.dskname: DATA_0001 ; 0x028: length=9 kfdhdb.grpname: DATA ; 0x048: length=4 kfdhdb.fgname: DATA_0001 ; 0x068: length=9 kfdhdb.capname: ; 0x088: length=0 --> here we need to make sure that we pick up the right device before erasing data with dd ! [root@grac42 ~]# dd if=/dev/zero of=/dev/asmdisk1_udev_sdc1 bs=8192 count=1000 1000+0 records in 1000+0 records out 8192000 bytes (8.2 MB) copied, 0.0213478 s, 384 MB/s [root@grac42 ~]# kfed read /dev/asmdisk1_udev_sdc1 | egrep 'type|name' kfbh.type: 0 ; 0x002: KFBTYP_INVALID KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0] --> disk header erased Here is a good point to change the ASM disk name ( if needed ) and to add add. ASM devices Note: Always check all nodes in your cluster to be sure we pick up the right devices You may use kfed to see wether we get consistent results on all nodes [grid@grac41 ASM]$ ssh grac41 asmcmd lsdsk -k --candidate Total_MB Free_MB OS_MB Name Failgroup Failgroup_Type Library Label UDID Product Redund Path 0 0 10236 REGULAR System UNKNOWN /dev/asm_data_10g_disk1 0 0 10236 REGULAR System UNKNOWN /dev/asm_data_10g_disk2 0 0 1019 REGULAR System UNKNOWN /dev/asm_test_1G_disk1 0 0 1019 REGULAR System UNKNOWN /dev/asm_test_1G_disk2 [grid@grac41 ASM]$ ssh grac42 asmcmd lsdsk -k --candidate Total_MB Free_MB OS_MB Name Failgroup Failgroup_Type Library Label UDID Product Redund Path 0 0 10236 REGULAR System UNKNOWN /dev/asm_data_10g_disk1 0 0 10236 REGULAR System UNKNOWN /dev/asm_data_10g_disk2 0 0 1019 REGULAR System UNKNOWN /dev/asm_test_1G_disk1 0 0 1019 REGULAR System UNKNOWN /dev/asm_test_1G_disk2 [grid@grac41 ASM]$ ssh grac43 asmcmd lsdsk -k --candidate Total_MB Free_MB OS_MB Name Failgroup Failgroup_Type Library Label UDID Product Redund Path 0 0 10236 REGULAR System UNKNOWN /dev/asm_data_10g_disk1 0 0 10236 REGULAR System UNKNOWN /dev/asm_data_10g_disk2 0 0 1019 REGULAR System UNKNOWN /dev/asm_test_1G_disk1 0 0 1019 REGULAR System UNKNOWN /dev/asm_test_1G_disk2 --> /dev/asm_data_10g_disk1 and /dev/asm_data_10g_disk2 are available in all cluster nodes Use asmca and add disks and monitor progress and ASM disk status : SQL> select g.name disk_grp, o.group_number, operation , est_minutes from gv$asm_operation o, gv$asm_diskgroup g where g.group_number = o.group_number and o.inst_id = g.inst_id; DISK_GRP GROUP_NUMBER OPERA EST_MINUTES ------------------------------ ------------ ----- ----------- DATA 3 REBAL 28 DATA 3 REBAL DATA 3 REBAL SQL> select d.inst_id, dg.name dg_name, dg.state dg_state, dg.type,d.name, d.DISK_NUMBER dsk_no, d.MOUNT_STATUS, d.HEADER_STATUS, d.MODE_STATUS, d.STATE, d. PATH, d.FAILGROUP FROM GV$ASM_DISK d, gv$asm_diskgroup dg where dg.group_number(+)=d.group_number and d.inst_id = dg.inst_id and dg.name = 'DATA' order by d.inst_id, d.group_number ; INST_ID DG_NAME DG_STATE TYPE NAME DSK_NO MOUNT_S HEADER_STATU MODE_ST STATE PATH FAILGROUP ---------- ---------- ---------- ------ -------------------- ------- ------- ------------ ------- -------- ---------------------------------------- --------------- 1 DATA MOUNTED NORMAL DATA_0004 4 CACHED MEMBER ONLINE NORMAL /dev/asm_data_10g_disk2 DATA_0004 1 DATA MOUNTED NORMAL DATA_0002 2 CACHED MEMBER ONLINE NORMAL /dev/asmdisk1_udev_sdd1 DATA_0002 1 DATA MOUNTED NORMAL DATA_0003 3 CACHED MEMBER ONLINE NORMAL /dev/asmdisk1_udev_sde1 DATA_0003 1 DATA MOUNTED NORMAL _DROPPED_0001_DATA 1 MISSING UNKNOWN OFFLINE FORCING DATA_0001 1 DATA MOUNTED NORMAL DATA_0000 0 CACHED MEMBER ONLINE NORMAL /dev/asm_data_10g_disk1 DATA_0000 2 DATA MOUNTED NORMAL DATA_0004 4 CACHED MEMBER ONLINE NORMAL /dev/asm_data_10g_disk2 DATA_0004 2 DATA MOUNTED NORMAL _DROPPED_0001_DATA 1 MISSING UNKNOWN OFFLINE FORCING DATA_0001 2 DATA MOUNTED NORMAL DATA_0003 3 CACHED MEMBER ONLINE NORMAL /dev/asmdisk1_udev_sde1 DATA_0003 2 DATA MOUNTED NORMAL DATA_0002 2 CACHED MEMBER ONLINE NORMAL /dev/asmdisk1_udev_sdd1 DATA_0002 2 DATA MOUNTED NORMAL DATA_0000 0 CACHED MEMBER ONLINE NORMAL /dev/asm_data_10g_disk1 DATA_0000 3 DATA MOUNTED NORMAL DATA_0004 4 CACHED MEMBER ONLINE NORMAL /dev/asm_data_10g_disk2 DATA_0004 3 DATA MOUNTED NORMAL DATA_0000 0 CACHED MEMBER ONLINE NORMAL /dev/asm_data_10g_disk1 DATA_0000 3 DATA MOUNTED NORMAL _DROPPED_0001_DATA 1 MISSING UNKNOWN OFFLINE FORCING DATA_0001 3 DATA MOUNTED NORMAL DATA_0002 2 CACHED MEMBER ONLINE NORMAL /dev/asmdisk1_udev_sdd1 DATA_0002 3 DATA MOUNTED NORMAL DATA_0003 3 CACHED MEMBER ONLINE NORMAL /dev/asmdisk1_udev_sde1 DATA_0003
very helpful thanks for knowledge sharing