Debugging ACFS mount errors: ACFS-02017

Resource status after ACFS mount fails with ACFS-02017

[root@gract2 ~]#  mount -t acfs /dev/asm/acfs_vol1-443 /u01/acfs/acfs-vol1
mount.acfs: CLSU-00100: operating system function: open64 failed with error data: 2
mount.acfs: CLSU-00101: operating system error message: No such file or directory
mount.acfs: CLSU-00103: error location: OOF_1
mount.acfs: CLSU-00104: additional error information: open64 (/dev/asm/acfs_vol1-443)
mount.acfs: ACFS-02017: Failed to open volume /dev/asm/acfs_vol1-443. Verify the volume exists.

[root@gract1 var]# crs 
*****  Local Resources: *****
Rescource NAME                 TARGET     STATE           SERVER       STATE_DETAILS                       
-------------------------      ---------- ----------      ------------ ------------------                  
ora.ACFS_DG1.ACFS_VOL1.advm    ONLINE     ONLINE          gract1       Volume device /dev/a sm/acfs_vol1-443 isonline,STABLE
ora.ACFS_DG1.ACFS_VOL1.advm    ONLINE     OFFLINE         gract2       STABLE   
ora.ACFS_DG1.ACFS_VOL1.advm    ONLINE     ONLINE          gract3       Volume device /dev/a sm/acfs_vol1-443 isonline,STABLE

[grid@gract2 ~]$  asmcmd  volinfo --all
ASMCMD-9470: ASM proxy instance unavailable
ASMCMD-9473: volume STATE will show as REMOTE
Diskgroup Name: ACFS_DG1
	 Volume Name: ACFS_VOL1
	 Volume Device: UNKNOWN
	 State: DISABLED
	 Size (MB): 800
	 Resize Unit (MB): 32
	 Redundancy: HIGH
	 Stripe Columns: 4
	 Stripe Width (K): 128
	 Usage: ACFS
	 Mountpath: /u01/acfs/acfs-vol1

If your proxy server is up and running Enable the Volumne by running an try to mount your ACFS fs
ASMCMD> volenable -a

If your proxy server is not running continue reading
[root@gract1 Desktop]#  crs | egrep 'advm|acfs|--|STAT'
Rescource NAME                 TARGET     STATE           SERVER       STATE_DETAILS                       
-------------------------      ---------- ----------      ------------ ------------------                  
ora.ACFS_DG1.ACFS_VOL1.advm    ONLINE     ONLINE          gract1       Volume device /dev/a sm/acfs_vol1-443 isonline,STABLE
ora.ACFS_DG1.ACFS_VOL1.advm    ONLINE     OFFLINE         gract2       STABLE   
ora.ACFS_DG1.ACFS_VOL1.advm    ONLINE     ONLINE          gract3       Volume device /dev/a sm/acfs_vol1-443 isonline,STABLE
ora.acfs_dg1.acfs_vol1.acfs    ONLINE     ONLINE          gract1       mounted on /u01/acfs /acfs-vol1,STABLE
ora.acfs_dg1.acfs_vol1.acfs    ONLINE     OFFLINE         gract2       STABLE   
ora.acfs_dg1.acfs_vol1.acfs    ONLINE     ONLINE          gract3       mounted on /u01/acfs /acfs-vol1,STABLE
ora.proxy_advm                 ONLINE     ONLINE          gract1       STABLE   
ora.proxy_advm                 ONLINE     OFFLINE         gract2       STABLE   
ora.proxy_advm                 ONLINE     ONLINE          gract3       STABLE   
--> ADVM PROXY server on gract2 is OFFILINE 
    

Fix ADVM proxy server startup problem

 
Status
[root@gract1 Desktop]# srvctl start volume -volume ACFS_VOL1 -diskgroup ACFS_DG1
PRCR-1079 : Failed to start resource ora.ACFS_DG1.ACFS_VOL1.advm
CRS-5017: The resource action "ora.proxy_advm start" encountered the following error: 
ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/gract2/crs/trace/crsd_oraagent_grid.trc".
CRS-2674: Start of 'ora.proxy_advm' on 'gract2' failed

[root@gract2 ~]#  srvctl start asm -proxy
PRCR-1079 : Failed to start resource ora.proxy_advm
CRS-5017: The resource action "ora.proxy_advm start" encountered the following error: 
ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/gract2/crs/trace/crsd_oraagent_grid.trc".
CRS-2674: Start of 'ora.proxy_advm' on 'gract2' failed
--> Both starting ASM proxy instance and starting the ACFS volume fails with CRS-5017, ORA-3113

Checking alert.log of +APX2 instance
Errors in file /u01/app/grid/diag/apx/+apx/+APX2/trace/+APX2_vdbg_25183.trc:
ORA-15477: cannot communicate with the volume driver
USER (ospid: 25183): terminating the instance due to error 15477

VDBG process from +APX" instance reports followin eror 
+APX2_vdbg_25183.trc: 
Function = kfvOpen, opname = Open syscall failed, location = skvolopen_1 
	 errbuf = 13, 
	 msgbuf = Permission denied 
	Other info = Filename:/dev/asm/.asm_ctl_vdbg

[root@gract2 ~]# ls -l /dev/asm/.asm_ctl_vdbg
brw-------. 1 root root 251, 1 Aug  9 14:04 /dev/asm/.asm_ctl_vdbg

--> File has a wrong protection 

Changing protections to match protections on gract1 and gract3 and restart ASM PROXY instance
[root@gract2 trace]# chmod 770  /dev/asm/.asm*
[root@gract2 trace]# chown root:asmadmin /dev/asm/.asm*
[root@gract2 trace]# ls -l /dev/asm/.asm*
brwxrwx---. 1 root asmadmin 251,  0 Aug  9 14:04 /dev/asm/.asm_ctl_spec
brwxrwx---. 1 root asmadmin 251, 10 Aug  9 14:04 /dev/asm/.asm_ctl_vbg0
brwxrwx---. 1 root asmadmin 251, 11 Aug  9 14:04 /dev/asm/.asm_ctl_vbg1
brwxrwx---. 1 root asmadmin 251, 12 Aug  9 14:04 /dev/asm/.asm_ctl_vbg2
brwxrwx---. 1 root asmadmin 251, 13 Aug  9 14:04 /dev/asm/.asm_ctl_vbg3
brwxrwx---. 1 root asmadmin 251, 14 Aug  9 14:04 /dev/asm/.asm_ctl_vbg4
brwxrwx---. 1 root asmadmin 251, 15 Aug  9 14:04 /dev/asm/.asm_ctl_vbg5
brwxrwx---. 1 root asmadmin 251, 16 Aug  9 14:04 /dev/asm/.asm_ctl_vbg6
brwxrwx---. 1 root asmadmin 251, 17 Aug  9 14:04 /dev/asm/.asm_ctl_vbg7
brwxrwx---. 1 root asmadmin 251, 18 Aug  9 14:04 /dev/asm/.asm_ctl_vbg8
brwxrwx---. 1 root asmadmin 251,  1 Aug  9 14:04 /dev/asm/.asm_ctl_vdbg
brwxrwx---. 1 root asmadmin 251,  3 Aug  9 14:04 /dev/asm/.asm_ctl_vio
brwxrwx---. 1 root asmadmin 251,  4 Aug  9 14:04 /dev/asm/.asm_ctl_vio0
brwxrwx---. 1 root asmadmin 251,  5 Aug  9 14:04 /dev/asm/.asm_ctl_vio1
brwxrwx---. 1 root asmadmin 251,  6 Aug  9 14:04 /dev/asm/.asm_ctl_vio2
brwxrwx---. 1 root asmadmin 251,  7 Aug  9 14:04 /dev/asm/.asm_ctl_vio3
brwxrwx---. 1 root asmadmin 251,  2 Aug  9 14:04 /dev/asm/.asm_ctl_vmb

[root@gract2 gpnpd]#  srvctl start asm -proxy
ADVM proxy instance starts sucessfully !
[root@gract2 gpnpd]# srvctl status asm -proxy
--> ADVM proxy is running on node gract3,gract2,gract1
[root@gract2 gpnpd]# crs
*****  Local Resources: *****
Rescource NAME                 TARGET     STATE           SERVER       STATE_DETAILS                       
-------------------------      ---------- ----------      ------------ ------------------                  
ora.ACFS_DG1.ACFS_VOL1.advm    ONLINE     ONLINE          gract1       Volume device /dev/a sm/acfs_vol1-443 isonline,STABLE
ora.ACFS_DG1.ACFS_VOL1.advm    ONLINE     ONLINE          gract2       Volume device /dev/a sm/acfs_vol1-443 isonline,STABLE
ora.ACFS_DG1.ACFS_VOL1.advm    ONLINE     ONLINE          gract3       Volume device /dev/a sm/acfs_vol1-443 isonline,STABLE
ora.acfs_dg1.acfs_vol1.acfs    ONLINE     ONLINE          gract1       mounted on /u01/acfs /acfs-vol1,STABLE
ora.acfs_dg1.acfs_vol1.acfs    ONLINE     OFFLINE         gract2       STABLE   
ora.acfs_dg1.acfs_vol1.acfs    ONLINE     ONLINE          gract3       mounted on /u01/acfs /acfs-vol1,STABLE

Fixing ACFS mount error : ACFS-02017

Both mount and  srvctl start filesystem fails with ACFS-2017
[root@gract1 bin]# srvctl start filesystem -device  /dev/asm/acfs_vol1-443 -node gract2
PRCR-1013 : Failed to start resource ora.acfs_dg1.acfs_vol1.acfs
PRCR-1064 : Failed to start resource ora.acfs_dg1.acfs_vol1.acfs on node gract2
CRS-5016: Process "/bin/mount" spawned by agent "ORAROOTAGENT" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/grid/diag/crs/gract2/crs/trace/crsd_orarootagent_root.trc"
Error mounting file system '/dev/asm/acfs_vol1-443' on '/u01/acfs/acfs-vol1': mount.acfs: CLSU-00100: operating system function: open64 failed with error data: 1
mount.acfs: CLSU-00101: operating system error message: Operation not permitted
mount.acfs: CLSU-00103: error location: OOF_1
mount.acfs: CLSU-00104: additional error information: open64 (/dev/asm/acfs_vol1-443)
mount.acfs: ACFS-02017: Failed to open volume /dev/asm/acfs_vol1-443. Verify the volume exists.
CRS-2674: Start of 'ora.acfs_dg1.acfs_vol1.acfs' on 'gract2' failed

Checking protection of mountpoint and ACFS device  
Note if this file doesn't exit you can use mknod to recreate that file 
[root@gract2 trace]# ls -l /dev/asm/acfs_vol1-443 
brw-------. 1 root root 251, 226817 Aug 12 17:25 /dev/asm/acfs_vol1-443

root@gract2 trace]# chmod 770 /dev/asm/acfs_vol1-443
[root@gract2 trace]# chown root:asmadmin  /dev/asm/acfs_vol1-443
[root@gract2 trace]# ls -l  /dev/asm/acfs_vol1-443
brwxrwx---. 1 root asmadmin 251, 226817 Aug 12 17:25 /dev/asm/acfs_vol1-443

Even after all file protections are matching protections on remote node - ACFS  still get mount error

Trying to reload ACFS driver 
[root@gract2 trace]# crsctl stop crs 
[root@gract2 trace]# acfsroot install
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9118: oracleadvm.ko driver in use - cannot unload.
FATAL: Module oracleadvm is in use.
ACFS-9119: oracleadvm.ko driver failed to unload.
ACFS-9427: Failed to unload ADVM/ACFS drivers. A system reboot is recommended.
[root@gract2 trace]# /sbin/lsmod | grep oracle
oracleacfs           3310765  0 
oracleadvm            506070  3 
oracleoks             508392  2 oracleacfs,oracleadvm
oracleasm              53865  1 
--> Problem still there: ACFS mount throws   ACFS-02017

Final Fix : Removing /var/tmp/.oracle and reboot CRS stack
For details read  
  Bug 13578742 : ACFS IS NOT MOUNTING WITH CLSU-00101: OPERATING SYSTEM ERROR MESSAGE: OPERATION
[root@gract2 trace]#  rm -rf /var/tmp/.oracle/*
[root@gract2 trace]#  ls -l  /var/tmp/.oracle/*
ls: cannot access /var/tmp/.oracle/*: No such file or directory
[root@gract2 trace]# reboot
[root@gract2 ~]#   crs  | egrep 'acfs|proxy|ACFS|NAME|--' | egrep 'gract2|NAME|--'
Rescource NAME                 TARGET     STATE           SERVER       STATE_DETAILS                       
-------------------------      ---------- ----------      ------------ ------------------                  
ora.ACFS_DG1.ACFS_VOL1.advm    ONLINE     ONLINE          gract2       Volume device /dev/a sm/acfs_vol1-443 isonline,STABLE
ora.ACFS_DG1.dg                ONLINE     ONLINE          gract2       STABLE   
ora.acfs_dg1.acfs_vol1.acfs    ONLINE     ONLINE          gract2       mounted on /u01/acfs /acfs-vol1,STABLE
ora.proxy_advm                 ONLINE     ONLINE          gract2       STABLE 
[root@gract2 ~]# mount | grep acfs
/dev/asm/acfs_vol1-443 on /u01/acfs/acfs-vol1 type acfs (rw)

Reference

  • Bug 13578742 : ACFS IS NOT MOUNTING WITH CLSU-00101: OPERATING SYSTEM ERROR MESSAGE: OPERATION
  • ACFS Filesystem Cannot Be Mounted On RAC Due To ACFS-02017. (Doc ID 1440385.1)

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>