ORA-15040, ORA-15042 errors mounting a diskgroup

Check current status

Try manually mount diskgroup
ASMCMD> mount -a
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "1" is missing from group number "1" 

Check ASM alert.log
SQL> alter diskgroup ACFS mount 
NOTE: cache registered group ACFS number=1 incarn=0x09884abc
NOTE: cache began mount (first) of group ACFS number=1 incarn=0x09884abc
NOTE: Assigning number (1,0) to disk (/dev/oracleasm/disks/ACFS_DATA)
Mon Aug 19 10:29:24 2013
NOTE: GMON heartbeating for grp 1
GMON querying group 1 at 334 for pid 31, osid 8882
NOTE: Assigning number (1,1) to disk ()
GMON querying group 1 at 335 for pid 31, osid 8882
NOTE: cache dismounting (clean) group 1/0x09884ABC (ACFS) 
NOTE: messaging CKPT to quiesce pins Unix process pid: 8882, image: oracle@grac1.example.com (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 1/0x09884ABC (ACFS) 
NOTE: cache ending mount (fail) of group ACFS number=1 incarn=0x09884abc
NOTE: cache deleting context for group ACFS 1/0x09884abc
GMON dismounting group 1 at 336 for pid 31, osid 8882
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
ERROR: diskgroup ACFS was not mounted
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "1" is missing from group number "1" 
ERROR: alter diskgroup ACFS mount
Mon Aug 19 10:29:27 2013
ASM Health Checker found 1 new failures

Display even dismounted diskgroups
$  asmcmd lsdg --discovery
State       Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
DISMOUNTED          N           0   4096        0         0        0                0               0              0             N  ACFS/
MOUNTED     NORMAL  N         512   4096  1048576     15342     9932             5114            2409              0             N  DATA/
MOUNTED     NORMAL  N         512   4096  1048576      6141     5217             2047            1585              0             Y  OCR/
--> diskgroup ACFS is still dismounted  
Try to mount the ASM diskgroup with force option and check the available disks
As we have only a single disk available for NORMAL redundancy the mount fails and we need to use the force option  
SQL> alter diskgroup ACFS mount force;
Diskgroup altered.

Verify the disk status after mount force command. 
$ asmcmd lsdsk -p -G ACFS
Group_Num  Disk_Num      Incarn  Mount_Stat  Header_Stat  Mode_Stat  State   Path
        1         1  3915954601  MISSING     UNKNOWN      OFFLINE    NORMAL  
        1         0  3915954600  CACHED      MEMBER       ONLINE     NORMAL  /dev/oracleasm/disks/ACFS_DATA
$ asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512   4096  1048576      1019      122                0              61              1             N  ACFS/
MOUNTED  NORMAL  N         512   4096  1048576     15342     9932             5114            2409              0             N  DATA/
MOUNTED  NORMAL  N         512   4096  1048576      6141     5217             2047            1585              0             Y  OCR/
           1585              0             Y  OCR/
Try to read and verify ASM disk header ( on all instances )
# $GRID_HOME/bin/kfed read  /dev/sdj1
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
7FB74EC83400 00000000 00000000 00000000 00000000  [................]
        Repeat 63 times
7FB74EC83800 0000FF00 0003FBB8 000032FC 0003DA6D  [.........2..m...]
7FB74EC83810 0000FEF5 00000000 00000002 00000002  [................]
7FB74EC83940 00000000 00000000 00000000 01000000  [................]
7FB74EC83950 00000000 00000000 00000000 001C001C  [................]
7FB74EC83960 00000001 00000000 00000000 00000000  [................]
7FB74EC83970 00000000 00000004 00008196 00000000  [................]
7FB74EC83980 00000000 00000000 00000000 00000000  [................]
  Repeat 167 times
KFED-00322: file not found; arguments: [kfbtTraverseBlock] [Invalid OSM block type] [] [0]
--> This is not a valid  ASM header - let's create new ASM disk:

Fix the problem by recreating the ASM disk

Check device numbers so we get a link between ASM  devices and OS devices
# ls -l  /dev/sdj1
brw-rw---- 1 root disk 8, 145 Aug 19 09:56 /dev/sdj1
# ls -l  /dev/sdi1
brw-rw---- 1 root disk 8, 129 Aug 19 09:56 /dev/sdi1
#  ls -l /dev/oracleasm/disks/ACFS_DATA
brw-rw---- 1 grid asmadmin 8, 129 Aug 19 10:48 /dev/oracleasm/disks/ACFS_DATA

Create new asm diks for /dev/sdj1 with device numbers 8, 145
# /etc/init.d/oracleasm createdisk acfs_data2 /dev/sdj1
Marking disk "acfs_data2" as an ASM disk:                  [  OK  ]

# /etc/init.d/oracleasm  listdisks
ACFS_DATA
ACFS_DATA2


Let's add the new ASM disk to our diskgroup
SQL> alter diskgroup ACFS add disk '/dev/oracleasm/disks/ACFS_DATA2';
alter diskgroup ACFS add disk '/dev/oracleasm/disks/ACFS_DATA2'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15075: disk(s) are not visible cluster-wide

--> Verify that  we can read the ASM header files on each instance by running 
$GRID_HOME/bin/kfed read   /dev/oracleasm/disks/ACFS_DATA2 
and check that the related files is accessible at OS level.

After that add the new disk to our diskgroup  
SQL> alter diskgroup ACFS add disk  '/dev/oracleasm/disks/ACFS_DATA2';
$ asmcmd lsdsk -p -G ACFS
Group_Num  Disk_Num      Incarn  Mount_Stat  Header_Stat  Mode_Stat  State   Path
        1         1  3915954605  MISSING     UNKNOWN      OFFLINE    NORMAL  
        1         0  3915954604  CACHED      MEMBER       ONLINE     NORMAL  /dev/oracleasm/disks/ACFS_DATA
        1         3  3915954610  CACHED      MEMBER       ONLINE     NORMAL  /dev/oracleasm/disks/ACFS_DATA2

Drop the disk with Mount_Status : MISSING and Header_Status: UNKNOWN from the diskgroup.
As we don't have the the Disk Path we need  drop the related failgroup
SQL> select DISK_NUMBER, MOUNT_STATUS, HEADER_STATUS, MODE_STATUS, FAILGROUP, PATH from  v$asm_disk; 
DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST FAILGROUP   PATH 
----------- ------- ------------ ------- ----------- --------------------------------------------------------------------------------
      1 MISSING UNKNOWN     OFFLINE ACFS_0001
      3 CACHED  MEMBER     ONLINE  ACFS_0003   /dev/oracleasm/disks/ACFS_DATA2

SQL> alter diskgroup ACFS drop disks in failgroup ACFS_0001 force;
Diskgroup altered.
Verify the drop operation worked successfully
SQL> !asmcmd lsdsk -p -G ACFS
Group_Num  Disk_Num      Incarn  Mount_Stat  Header_Stat  Mode_Stat  State    Path
        1         1  3915954605  MISSING     UNKNOWN      OFFLINE    FORCING  
        1         0  3915954604  CACHED      MEMBER       ONLINE     NORMAL   /dev/oracleasm/disks/ACFS_DATA
        1         3  3915954610  CACHED      MEMBER       ONLINE     NORMAL   /dev/oracleasm/disks/ACFS_DATA2
After some seconds  drop operation should succeed :
SQL> !asmcmd lsdsk -p -G ACFS
Group_Num  Disk_Num      Incarn  Mount_Stat  Header_Stat  Mode_Stat  State   Path
        1         0  3915954604  CACHED      MEMBER       ONLINE     NORMAL  /dev/oracleasm/disks/ACFS_DATA
        1         3  3915954610  CACHED      MEMBER       ONLINE     NORMAL  /dev/oracleasm/disks/ACFS_DATA2

Note if any ASM disk is corrupted you may need to clean the ASM header first  by running
 dd if=/dev/zero  of=/dev/sdi1 bs=8192  count=1000

But be very cautious as selecting a wrong device will lead to a complete data loss !

References :

2 thoughts on “ORA-15040, ORA-15042 errors mounting a diskgroup

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>