Overview ACFS/HANFS
- HANFS 12.1 only supports NFS v3 over Ipv4, with no NFS locking
- HANFS requires NFS in order to run.
- It is assumed that NFS (and its associated needs) will be started by init scripts at node boot time.
- NFS will need to be running on all nodes of the cluster.
- The portmap service was used to map RPC program numbers to IP address port number combinations in earlier versions of Red Hat Enterprise Linux.
- As per RHEL6 docs, portmap service has been replaced by rpcbind in Red Hat Enterprise Linux 6 to enable IPv6 support.
Setup NFS ( on all RAC nodes )
Start and configure NFS
[root@gract2 ~]# service nfs status
rpc.svcgssd is stopped
rpc.mountd is stopped
nfsd is stopped
rpc.rquotad is stopped
[root@gract2 ~]# service nfs start
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS mountd: [ OK ]
Stopping RPC idmapd: [ OK ]
Starting RPC idmapd: [ OK ]
Starting NFS daemon: [ OK ]
[root@gract2 ~]# service rpcbind start
[root@gract2 ~]# chkconfig nfs on
[root@gract2 ~]# chkconfig rpcbind on
Setup ACFS and create a static HANFS VIP host in your DNS
Configure a static DNS address
[root@gract1 var]# nslookup nfsvip
Server: 192.168.1.50
Address: 192.168.1.50#53
Name: nfsvip.example.com
Address: 192.168.1.199
Create an ACFS filessytem ( please read following node to continue )
Verify ACFS resources
[root@gract1 Desktop]# crs | egrep 'acfs|advm|proxy|ACFS|NAME|---'
Rescource NAME TARGET STATE SERVER STATE_DETAILS
------------------------- ---------- ---------- ------------ ------------------
ora.ACFS_DG1.ACFS_VOL1.advm ONLINE ONLINE gract1 Volume device /dev/a sm/acfs_vol1-443 isonline,STABLE
ora.ACFS_DG1.ACFS_VOL1.advm ONLINE ONLINE gract2 Volume device /dev/a sm/acfs_vol1-443 isonline,STABLE
ora.ACFS_DG1.ACFS_VOL1.advm ONLINE ONLINE gract3 Volume device /dev/a sm/acfs_vol1-443 isonline,STABLE
ora.ACFS_DG1.dg ONLINE ONLINE gract1 STABLE
ora.ACFS_DG1.dg ONLINE ONLINE gract2 STABLE
ora.ACFS_DG1.dg ONLINE ONLINE gract3 STABLE
ora.acfs_dg1.acfs_vol1.acfs ONLINE ONLINE gract1 mounted on /u01/acfs /acfs-vol1,STABLE
ora.acfs_dg1.acfs_vol1.acfs ONLINE ONLINE gract2 mounted on /u01/acfs /acfs-vol1,STABLE
ora.acfs_dg1.acfs_vol1.acfs ONLINE ONLINE gract3 mounted on /u01/acfs /acfs-vol1,STABLE
ora.proxy_advm ONLINE ONLINE gract1 STABLE
ora.proxy_advm ONLINE ONLINE gract2 STABLE
ora.proxy_advm ONLINE ONLINE gract3 STABLE
To continue HANFS setup following resource needs to be ONLINE on all RAC nods
- DG where our ACFS Volumne sits on top ora.ACFS_DG1.dg
- PROXY ADVM instance to handle ACFS operations ora.proxy_advm
- ACFS Volumne manager
- ACFS files system
In case you get an error Debugging ACFS mount errors : ACFS-02017 please read following note.
Creating the HAVIP and Exportfs cluster resource
Setup a static HANFS VIP host in your DNS and verifiy this address
[root@gract1 var]# nslookup nfsvip
Server: 192.168.1.50
Address: 192.168.1.50#53
Name: nfsvip.example.com
Address: 192.168.1.199
Note HAVIP ID and exportfs name should be lower case to avoid CRS-2674, PRCR-1079 error during staring the HAVIP
Create HAVIP resource
[root@gract1 grid]# srvctl add havip -id havip_id -address nfsvip.example.com -netnum 1 -description "Helmut's XXXX Project"
[root@gract1 grid]# crs | egrep 'hanfs|havip|export|NAME|---'
Resource NAME INST TARGET STATE SERVER STATE_DETAILS
--------------------------- ---- ------------ ------------ --------------- -----------------------------------------
ora.havip_id.havip 1 OFFLINE OFFLINE - STABLE
Trying to bring this resource ONLINE
[root@gract1 Desktop]# srvctl start havip -id havip_id
PRCE-1026 : Cannot start HAVIP resource without an Export FS resource.
Unable to start 'ora.hr1.havip' because it has a 'hard' dependency on resource type 'ora.HR1.export.type' and no resource of
that type can satisfy the dependency
[root@gract1 Desktop]# crsctl status resource ora.hr.havip -f | grep START_DEPENDENCIES
START_DEPENDENCIES=hard(ora.net1.network,uniform:type:ora.HR.export.type) weak(global:ora.gns) dispersion:active(type:ora.havip.type)
pullup(ora.net1.network) pullup:always(type:ora.HR.export.type)
Why the failure to start?
Recall earlier that we mentioned that an HAVIP requires1 or more ExportFS configured.
Without an ExportFS, the HAVIP will not start. If a client had mounted the ExportFS and the HAVIP started without the ExportFS
available, the client would receive an ESTALE error. This resource dependency prevents the resumption of NFS services on the
client until the server side file system is available for access.
Create exportfs cluster rescource
[root@gract1 grid]# srvctl add exportfs -id havip_id -path /u01/acfs/acfs-vol1 -name hanfs -options "rw,no_root_squash"
[root@gract1 grid]# crs | egrep 'acfs|hhanfs|havip|export|NAME|---'
Rescource NAME TARGET STATE SERVER STATE_DETAILS
------------------------- ---------- ---------- ------------ ------------------
ora.ACFS_DG1.ACFS_VOL1.advm ONLINE ONLINE gract1 Volume device /dev/a sm/acfs_vol1-443 isonline,STABLE
ora.ACFS_DG1.ACFS_VOL1.advm ONLINE ONLINE gract2 Volume device /dev/a sm/acfs_vol1-443 isonline,STABLE
ora.ACFS_DG1.ACFS_VOL1.advm ONLINE ONLINE gract3 Volume device /dev/a sm/acfs_vol1-443 isonline,STABLE
ora.acfs_dg1.acfs_vol1.acfs ONLINE ONLINE gract1 mounted on /u01/acfs /acfs-vol1,STABLE
ora.acfs_dg1.acfs_vol1.acfs ONLINE ONLINE gract2 mounted on /u01/acfs /acfs-vol1,STABLE
ora.acfs_dg1.acfs_vol1.acfs ONLINE ONLINE gract3 mounted on /u01/acfs /acfs-vol1,STABLE
Resource NAME INST TARGET STATE SERVER STATE_DETAILS
--------------------------- ---- ------------ ------------ --------------- -----------------------------------------
ora.hanfs.export 1 OFFLINE OFFLINE - STABLE
ora.havip_id.havip 1 OFFLINE OFFLINE - STABLE
Start HAVIP
[root@gract1 Desktop]# srvctl start havip -id havip_id
Check status:
[root@gract1 grid]# srvctl status exportfs -id havip_id
export file system hanfs is enabled
export file system hanfs is exported on node gract2
--> HANFS fs is exported on gract2
[root@gract1 grid]# srvctl status exportfs -id havip_id
export file system hanfs is enabled
export file system hanfs is exported on node gract2
[root@gract1 grid]# showmount -e gract2
Export list for gract2:
/u01/acfs/acfs-vol1 192.168.1.0/24
[root@gract1 grid]# ssh gract2 exportfs -v
/u01/acfs/acfs-vol1
192.168.1.0/24(rw,wdelay,no_root_squash,no_subtree_check,fsid=1702682800)
[root@gract1 grid]# srvctl config exportfs -name hanfs
export file system hanfs is configured
Exported path: /u01/acfs/acfs-vol1
Export Options: rw,no_root_squash
Configured Clients: 192.168.1.0/24
Export file system is enabled.
Export file system is individually enabled on nodes:
Export file system is individually disabled on nodes:
Testing application failover with HAVIP
Login to our name server ( In that case NS is not a RAC instance )
Figure out our HAVIP address ( run the command below an any of yozur HUB nodes
[root@gract3 ~]# srvctl config havip
HAVIP exists: /havip_id/192.168.1.199, network number 1
Description: Helmut's XXXX Project
Home Node:
HAVIP is enabled.
HAVIP is individually enabled on nodes:
HAVIP is individually disabled on nodes:
Login to our name server or any other client
Create mount point
[root@ns1 ~]# chmod 755 hr
[root@ns1 ~]# chmod 755 /hr
[root@ns1 ~]# showmount -e gract2
Export list for gract2:
/u01/acfs/acfs-vol1 192.168.1.0/24
NFS mount that filesystem
[root@ns1 ~]# mount -t nfs nfsvip.example.com:/u01/acfs/acfs-vol1 /hr
[root@ns1 ~]# mount -t nfs
nfsvip.example.com:/u01/acfs/acfs-vol1 on /hr type nfs (rw,vers=4,addr=192.168.1.199,clientaddr=192.168.1.50)
Open a file in and editor and keep it open
[root@ns1 ~]# vi /hr/test_file
[root@gract2 ~]# cat /u01/acfs/acfs-vol1/test_file
This is my first line
Resource NAME INST TARGET STATE SERVER STATE_DETAILS
--------------------------- ---- ------------ ------------ --------------- -----------------------------------------
ora.hanfs.export 1 ONLINE ONLINE gract2 STABLE
ora.havip_id.havip 1 ONLINE ONLINE gract2 STABLE
--> HANFS fs is currently servered by node gract2
Failover Test I : Stop/crash node gract2 by a reboot
Resource NAME INST TARGET STATE SERVER STATE_DETAILS
--------------------------- ---- ------------ ------------ --------------- -----------------------------------------
ora.hanfs.export 1 ONLINE ONLINE gract2 STOPPING
ora.havip_id.havip 1 ONLINE OFFLINE - STABLE
--> Both CW resources exportfs and havip are stopping and failing over to gract3
Resource NAME INST TARGET STATE SERVER STATE_DETAILS
--------------------------- ---- ------------ ------------ --------------- -----------------------------------------
ora.hanfs.export 1 ONLINE ONLINE gract3 STABLE
ora.havip_id.havip 1 ONLINE ONLINE gract3 STABLE
Add a new line to our file and display that line from node gract3
[root@gract3 ~]# cat /u01/acfs/acfs-vol1/test_file
This is my first line
This is my second line after reboot of gract2
--> File sucessfully updated - no interuption despite the failover
Failover Test II : Relocate HAVIP
[root@gract3 ~]# srvctl relocate havip -id havip_id -n gract1 -f
HAVIP was relocated successfully
[root@gract3 ~]# crs | egrep 'acfshanfs|havip|export|NAME|---'
Rescource NAME TARGET STATE SERVER STATE_DETAILS
------------------------- ---------- ---------- ------------ ------------------
Resource NAME INST TARGET STATE SERVER STATE_DETAILS
--------------------------- ---- ------------ ------------ --------------- -----------------------------------------
ora.hanfs.export 1 ONLINE ONLINE gract1 STABLE
ora.havip_id.havip 1 ONLINE ONLINE gract1 STABLE
Update the file again and test application continuity from RAC node gract1
[root@gract1 Desktop]# cat /u01/acfs/acfs-vol1/test_file
This is my first line
This is my second line after reboot of gract2
This is my last line after relocation to gract1
Deleting exportfs and HAVIP resource
# srvctl stop exportfs -name hanfs -f
# srvctl remove exportfs -name hanfs
# srvctl remove havip -id havip_id
---> This command dumps core but HAVIP resource was deleted sucessfully
Reference