Debugging ACFS mount errors: ACFS-02017

Fixing ACFS mount error : ACFS-02017

Both mount and  srvctl start filesystem fails with ACFS-2017
[root@gract1 bin]# srvctl start filesystem -device  /dev/asm/acfs_vol1-443 -node gract2
PRCR-1013 : Failed to start resource ora.acfs_dg1.acfs_vol1.acfs
PRCR-1064 : Failed to start resource ora.acfs_dg1.acfs_vol1.acfs on node gract2
CRS-5016: Process "/bin/mount" spawned by agent "ORAROOTAGENT" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/grid/diag/crs/gract2/crs/trace/crsd_orarootagent_root.trc"
Error mounting file system '/dev/asm/acfs_vol1-443' on '/u01/acfs/acfs-vol1': mount.acfs: CLSU-00100: operating system function: open64 failed with error data: 1
mount.acfs: CLSU-00101: operating system error message: Operation not permitted
mount.acfs: CLSU-00103: error location: OOF_1
mount.acfs: CLSU-00104: additional error information: open64 (/dev/asm/acfs_vol1-443)
mount.acfs: ACFS-02017: Failed to open volume /dev/asm/acfs_vol1-443. Verify the volume exists.
CRS-2674: Start of 'ora.acfs_dg1.acfs_vol1.acfs' on 'gract2' failed

Checking protection of mountpoint and ACFS device  
Note if this file doesn't exit you can use mknod to recreate that file 
[root@gract2 trace]# ls -l /dev/asm/acfs_vol1-443 
brw-------. 1 root root 251, 226817 Aug 12 17:25 /dev/asm/acfs_vol1-443

root@gract2 trace]# chmod 770 /dev/asm/acfs_vol1-443
[root@gract2 trace]# chown root:asmadmin  /dev/asm/acfs_vol1-443
[root@gract2 trace]# ls -l  /dev/asm/acfs_vol1-443
brwxrwx---. 1 root asmadmin 251, 226817 Aug 12 17:25 /dev/asm/acfs_vol1-443

Even after all file protections are matching protections on remote node - ACFS  still get mount error

Trying to reload ACFS driver 
[root@gract2 trace]# crsctl stop crs 
[root@gract2 trace]# acfsroot install
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9118: oracleadvm.ko driver in use - cannot unload.
FATAL: Module oracleadvm is in use.
ACFS-9119: oracleadvm.ko driver failed to unload.
ACFS-9427: Failed to unload ADVM/ACFS drivers. A system reboot is recommended.
[root@gract2 trace]# /sbin/lsmod | grep oracle
oracleacfs           3310765  0 
oracleadvm            506070  3 
oracleoks             508392  2 oracleacfs,oracleadvm
oracleasm              53865  1 
--> Problem still there: ACFS mount throws   ACFS-02017

Final Fix : Removing /var/tmp/.oracle and reboot CRS stack
For details read  
[root@gract2 trace]#  rm -rf /var/tmp/.oracle/*
[root@gract2 trace]#  ls -l  /var/tmp/.oracle/*
ls: cannot access /var/tmp/.oracle/*: No such file or directory
[root@gract2 trace]# reboot
[root@gract2 ~]#   crs  | egrep 'acfs|proxy|ACFS|NAME|--' | egrep 'gract2|NAME|--'
Rescource NAME                 TARGET     STATE           SERVER       STATE_DETAILS                       
-------------------------      ---------- ----------      ------------ ------------------                  
ora.ACFS_DG1.ACFS_VOL1.advm    ONLINE     ONLINE          gract2       Volume device /dev/a sm/acfs_vol1-443 isonline,STABLE
ora.ACFS_DG1.dg                ONLINE     ONLINE          gract2       STABLE   
ora.acfs_dg1.acfs_vol1.acfs    ONLINE     ONLINE          gract2       mounted on /u01/acfs /acfs-vol1,STABLE
ora.proxy_advm                 ONLINE     ONLINE          gract2       STABLE 
[root@gract2 ~]# mount | grep acfs
/dev/asm/acfs_vol1-443 on /u01/acfs/acfs-vol1 type acfs (rw)

Leave a Reply

Your email address will not be published. Required fields are marked *