How ASM fixes block corruption with NORMAL redundancy ?

Overview

ASM always use the primary AU for to read data. If the primary AU is corrupted then ASM will read the secondary AU. 
If the secondary AU is well then ASM tries to overwrite the corrupted primary AU using the secondary AU. If the 
corrupted primary AU is fixed then that AU will be the primary AU as always. If the corrupted primary AU can’t 
be overwritten then ASM tries to write the new AU to other location in the disk. If that write operation is successfully 
then that AU will be the new primary AU.

Prepare test case

Checking OS location for ASM AU
SQL> select PXN_KFFXP, -- physical extent number \
      XNUM_KFFXP, -- virtual extent number
      DISK_KFFXP, -- disk number
      AU_KFFXP,    -- allocation unit number
      decode(LXN_KFFXP,0,'Primary',1,'Secondary','header metadata') "AU type"
    from X$KFFXP
    where NUMBER_KFFXP=256 -- ASM file 272
    AND GROUP_KFFXP=4 -- group number 1
    order by 1;

SK_KFFXP   AU_KFFXP AU type
---------- ---------- ---------- ---------- ---------------
     0        0           0    144 Primary
     1        0           1    144 Secondary
     2        1           1    145 Primary
     3        1           0    145 Secondary
Summary 
--> Data block OFFset: 133
--> Database block size: 8k
--> ASM File number : 256
--> ASM DG : 4
--> AU size: 1Mbyte
--> ASM disks :  Disk# 0 :  /dev/asm_test_1G_disk1 - Disk# 1: /dev/asm_test_1G_disk2
--> ASM disk  /dev/asm_test_1G_disk2 is the Primary for AU 145 

Reading the Primary AU
[root@grac41 Desktop]#  dd if=/dev/asm_test_1G_disk2  bs=8k  count=1 skip=18565   | od -a
0017760 stx   A stx  bs   A   S   M   -   T   E   S   T soh ack   i   n

Reading Secondary AU 
[root@grac41 Desktop]#  dd if=/dev/asm_test_1G_disk1 bs=8k  count=1 skip=18565   | od -a
0017760 stx   A stx  bs   A   S   M   -   T   E   S   T soh ack   i   n

Erasing 8k block in our primary AU and verify deletiondd if=/dev/zero of=/dev/asm_test_1G_disk2  bs=8k  count=1 seek=18565 
1+0 records in
1+0 records out
8192 bytes (8.2 kB) copied, 0.0345691 s, 237 kB/s
#  dd if=/dev/asm_test_1G_disk2  bs=8k  count=1 skip=18565   | od -a
0000000 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul
*
1+0 records in
1+0 records out

SQL> select * from test_tab;
     N NAME
---------- ----------------
     1 ASM-TEST 
--> Data is still valid - Is primary block already fixed by ASM ?

ot@grac41 Desktop]#  dd if=/dev/zero of=/dev/asm_test_1G_disk2  bs=8k  count=1 seek=18565
1+0 records in
1+0 records out
8192 bytes (8.2 kB) copied, 0.0279776 s, 293 kB/s
--> Block is still corrupted 

Flush buffer cache and monitor Database alert.log 
SQL> alter system flush buffer_cache;
System altered.

SQL> select * from test_tab;
         N NAME
---------- ----------------
         1 ASM-TEST

Checking alert.log 
Mon Jul 14 18:09:25 2014
ALTER SYSTEM: Flushing buffer cache
Mon Jul 14 18:09:42 2014
Hex dump of (file 7, block 133) in trace file /u01/app/oracle/diag/rdbms/grac4/grac41/trace/grac41_ora_29037.trc
Corrupt block relative dba: 0x01c00085 (file 7, block 133)
Completely zero block found during multiblock buffer read
Reading datafile '+TEST/grac4/datafile/test_ts.256.852905863' for corruption at rdba: 0x01c00085 (file 7, block 133)
Read datafile mirror 'TEST_0001' (file 7, block 133) found same corrupt data (no logical check)
Read datafile mirror 'TEST_0000' (file 7, block 133) found valid data
Hex dump of (file 7, block 133) in trace file /u01/app/oracle/diag/rdbms/grac4/grac41/trace/grac41_ora_29037.trc
Repaired corruption at (file 7, block 133)
--> block fixed by reading data from secondary disk 'TEST_0000' (file 7, block 133)

Verify fix using dd
[root@grac41 Desktop]#  dd if=/dev/asm_test_1G_disk2  bs=8k  count=1 skip=18565   | od -a
0000200 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul
*
0017740 nul nul nul nul nul nul nul nul nul nul nul nul nul   , soh stx
0017760 stx   A stx  bs   A   S   M   -   T   E   S   T soh ack   i   n
1+0 records in
1+0 records out
8192 bytes (8.2 kB) copied0020000

Summary

 

  • ASM recovers automatically the primary AU if it is corrupted.
  • The secondary AU will not be used unless a disk fail occurs.
  • The secondary AU is used for recovering the primary AU.
  • If ASM can’t overwrite the primary AU it will write the new primary AU in other disk part.
  • ASM writes an entry in the alert log when a recovering process occurs.

Reference

Locate ASM AU blocks at OS level

Overview

  • Create a smal tablespace with 5 Mbyte only ( == 12 AU if using NORMAL redundancy )
  • You need to connect to your ASM instance to see data from  X$KFFXP table

Setup test scenario

SQL script to create tablespace
col   "File name" format A50
col "Tablespace name" format A15

connect / as sysdba 
drop  tablespace test_ts including contents;
create tablespace test_ts  datafile '+TEST' size 5m;
select f.FILE#, f.NAME "File name", t.NAME "Tablespace name" from V$DATAFILE f, V$TABLESPACE t where t.NAME='TEST_TS' and f.TS# = t.TS#;

connect scott/tiger 
create table test_tab (n number, name varchar2(16)) tablespace test_ts;
insert into test_tab values (1, 'ASM-TEST');
commit;
select ROWID, NAME from test_tab;

Check and verify ASM related data

SQL Script :
connect cott/tiger@grac41
@get_rowid

connect sys/sys@grac41 as sysdba
show parameter db_block_size
select f.FILE#, f.NAME "File name", t.NAME "Tablespace name" from V$DATAFILE f, V$TABLESPACE t where t.NAME='TEST_TS' and f.TS# = t.TS#;

connect / as sysasm 
select GROUP_NUMBER from V$ASM_DISKGROUP where NAME='TEST';
select VALUE from V$ASM_ATTRIBUTE where NAME='au_size' and GROUP_NUMBER=4;
select GROUP_NUMBER, DISK_NUMBER, NAME, path from V$ASM_DISK  where  GROUP_NUMBER=4;

select PXN_KFFXP, -- physical extent number \
  XNUM_KFFXP, -- virtual extent number
  DISK_KFFXP, -- disk number
  AU_KFFXP    -- allocation unit number
from X$KFFXP
where NUMBER_KFFXP=256 -- ASM file 272
AND GROUP_KFFXP=4 -- group number 1
order by 1;

Output:
SQL> @check_it
Connected.
Rowid : AAAXgDAAHAAAACFAAA   Block No:133
PL/SQL procedure successfully completed.
--> Data block Offset: 133 

NAME                     TYPE     VALUE
------------------------------------ ----------- ------------------------------
db_block_size                 integer     8192
--> Database block size: 8k

     FILE# File name                          Tablespace name
---------- -------------------------------------------------- ---------------
     7 +TEST/grac4/datafile/test_ts.256.852905863          TEST_TS
--> ASM File number : 256

GROUP_NUMBER
------------
       4
--> ASM DG : 4

VALUE
------------------------------------------------------------------------------------------------------------------------------------
1048576
--> AU size: 1Mbyte
GROUP_NUMBER DISK_NUMBER NAME                PATH
------------ ----------- ------------------------------ ------------------------------
       4           0 TEST_0000            /dev/asm_test_1G_disk1
       4           1 TEST_0001            /dev/asm_test_1G_disk2
--> ASM disks :  Disk# 0 :  /dev/asm_test_1G_disk1 - Disk# 1: /dev/asm_test_1G_disk2

Summary of already collected data:
--> Data block OFFset: 133
--> Database block size: 8k
--> ASM File number : 256
--> ASM DG : 4
--> AU size: 1Mbyte
--> ASM disks :  Disk# 0 :  /dev/asm_test_1G_disk1 - Disk# 1: /dev/asm_test_1G_disk2

Mapping between AUs and OS Files
select PXN_KFFXP, -- physical extent number 
  XNUM_KFFXP, -- virtual extent number
  DISK_KFFXP, -- disk number
  AU_KFFXP    -- allocation unit number
from X$KFFXP
where NUMBER_KFFXP=256 -- ASM file 256 
AND GROUP_KFFXP=4 -- group number 1
order by 1;
 PXN_KFFXP XNUM_KFFXP DISK_KFFXP   AU_KFFXP
---------- ---------- ---------- ----------
     0        0           0    144
     1        0           1    144
     2        1           1    145
     3        1           0    145
     4        2           0    146
     5        2           1    146
     6        3           1    147
     7        3           0    147
     8        4           0    148
     9        4           1    148
    10        5           1    149
    11        5           0    149
12 rows selected.

--> As we have on 5 Myte we need 6 AUs  
    For normal reduncancy we need to double the AUs to 12 which matches perfect above print out 
    per AU we have 128 db blocks :  128 x 8k = 1Mbyte (== AU size )
    Our block number is 133 ---> our block is in the second AU ( XNUM_KFFXP = 1 ) at offset of 5 ( 133 - 128 = 5 )
    AU 1 is located at offset 145 MByte for disk 1 and disk 2 ( AU_KFFXP = 145 for DISK_KFFXP =1/2) 


Testing    that both disk already have written the block to disk:
[grid@grac41 Where_is_your_data]$  strings  /dev/asm_test_1G_disk1 | grep ASM-TEST
ASM-TEST
[grid@grac41 Where_is_your_data]$ strings  /dev/asm_test_1G_disk2 | grep ASM-TEST
ASM-TEST
--> Note you may need to wait until DBWR has written the data to disk 

Read a block of 1 Mbyte ( == AU size ) and  verify that we have picked up the right AU : disk offset 145 Mbyte
# dd if=/dev/asm_test_1G_disk1  bs=1024k count=1 skip=145 of=AU1_disk1
[grid@grac41 Where_is_your_data]$ strings AU1_disk1 | grep ASM-TEST
ASM-TEST
[grid@grac41 Where_is_your_data]$ od -c AU1_disk1
0137760 002 301 002  \b   A   S   M   -   T   E   S   T 001 006   i 356
Looks good - we have fount the correct AU

Extract Block 5 from AU[1] from disk  /dev/asm_test_1G_disk1 
[grid@grac41 Where_is_your_data]$   dd if=AU1_disk1 bs=8k count=1 skip=5 of=AU1_disk1_BLOCK_5
1+0 records in
1+0 records out
8192 bytes (8.2 kB) copied, 0.00119959 s, 6.8 MB/s
[grid@grac41 Where_is_your_data]$   strings AU1_disk1_BLOCK_5
ASM-TEST
--> We have found the correct block 

Calculate the disk offset -as this easily allows us to easily manipulate the specific block

--> Calculated 8k blocks Offset for partition start  : 
   (145*128 ) + 5 = 18565  ( 8Kb blocks )
   - 145 AU Offset ( if AU size = 1 Mbyte this translates to 128 8k blocks )
   -   5 block Offset in AU ( see above calculation )

Verify that our calculation is ok !
[grid@grac41 Where_is_your_data]$  dd if=/dev/asm_test_1G_disk1  bs=8k  count=1 skip=18565 of=block_disk1
1+0 records in
1+0 records out
8192 bytes (8.2 kB) copied, 0.00436751 s, 1.9 MB/s
[grid@grac41 Where_is_your_data]$ strings block_disk1
ASM-TEST
[grid@grac41 Where_is_your_data]$   dd if=/dev/asm_test_1G_disk2  bs=8k  count=1 skip=18565 of=block_disk2 
1+0 records in
1+0 records out
8192 bytes (8.2 kB) copied, 0.0110592 s, 741 kB/s
[grid@grac41 Where_is_your_data]$  strings block_disk1
ASM-TEST

Reference

Using kfed/dd to repair ASM disk header corruption

Overview

 

  • Note – Don’t run these commands on your production system as you can easily corrupt your data
  • In ASM versions 11.1.0.7 and later, the ASM disk header block is backed up in the second last ASM metadata block in the allocation unit 1.

Kfed parameters

  • aun – Allocation Unit (AU) number to read from. Default is AU0, or the very beginning of the ASM disk.
  • aus – AU size. Default is 1048576 (1MB). Specify the aus when reading from a disk group with non-default AU size.
  • blkn – block number to read. Default is block 0, or the very first block of the AU.
  • dev – ASM disk or device name. Note that the keyword dev can be omitted, but the ASM disk name is mandatory.