12c feature: FLEX ASM

Overview

  • Flex ASM can be configured for Standard Cluster and for Flex cluster   
  • Changing the cluster mode requires cluster downtime.   
  • Oracle does not support changing an Oracle Flex Cluster to an Oracle Clusterware standard Cluster.   
  • Oracle Flex Cluster requires Grid Naming Service (GNS).    Zone delegation is not required.

Verify ASM instance status

Check ASM cluster mode and state  
[grid@gract1 ~]$ asmcmd showclustermode
ASM cluster : Flex mode enabled

[grid@gract1 ~]$ asmcmd showclusterstate
Normal

[grid@gract1 ~/ASM]$   oifcfg getif
eth1  192.168.1.0  global  public
eth2  192.168.2.0  global  cluster_interconnect,asm
--> Only 2 networks present : ASM traffic uses standard interconnect

[grid@gract1 ~]$  olsnodes  -n -s -a
gract1    1    Active    Hub
gract2    2    Active    Hub
gract3    3    Active    Hub
--> Note:  only HUP nodes with access to storage can act as an ASM FLEX instance 

ASM resource status: 
[grid@gract1 ~]$  crs | egrep 'STATE|asm|--'
NAME                           TARGET     STATE           SERVER       STATE_DETAILS   
-------------------------      ---------- ----------      ------------ ------------------
ora.asm                        ONLINE     ONLINE          gract1       STABLE 
ora.asm                        ONLINE     ONLINE          gract3       STABLE 
ora.asm                        ONLINE     ONLINE          gract2       STABLE 

$  srvctl status asm -detail
ASM is running on gract3,gract2,gract1
ASM is enabled.

Testing ASM connection:

How many ASM connections do we expect ?
[grid@gract2 ASM]$ srvctl status asm -detail
ASM is running on gract2,gract1
ASM is enabled.

Let's connect to the global +ASM service
[grid@gract2 ASM]$  sqlplus sys/sys@gract1:/+ASM as sysasm @v
SQL> select INSTANCE_NUMBER,INSTANCE_NAME,STATUS, HOST_NAME    from v$instance;
INSTANCE_NUMBER INSTANCE_N STATUS    HOST_NAME
--------------- ---------- ------------ ------------------------------
          1 +ASM1       STARTED    gract1.example.com

[grid@gract2 ASM]$ sqlplus sys/sys@gract2:/+ASM as sysasm @v
SQL> select INSTANCE_NUMBER,INSTANCE_NAME,STATUS, HOST_NAME    from v$instance;
INSTANCE_NUMBER INSTANCE_N STATUS    HOST_NAME
--------------- ---------- ------------ ------------------------------
          2 +ASM2       STARTED    gract2.example.com

[grid@gract2 ASM]$  sqlplus sys/sys@gract3:/+ASM as sysasm 
ERROR:
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
--> This error is expected as +ASM servcice is not running on gract3

ASM Network

  • With Flex ASM Oracle 12c, a new type of network is called the ASM network.
  • This network is used for communication between ASM and its clients and is accessible on all the nodes. 
  • All ASM clients in the cluster have access to one or ore ASM network. 
  • Also it is possible to configure single network can perform both function as a private and an ASM network (oifcfg getif, you should see cluster_interconnect,asm)

ASM Listeners

  • To support FLex ASM, a set of ASM listeners are configured for every ASM network. 
  • Up to three ASM listener addresses are registred as remote listeners in each client database instance.
  • All clients connections are load balanced across the entire set of ASM instances
ASM Listener Details
Note: ASM listeners only run on nodes with ASM instances 
[grid@gract2 ~]$ srvctl  config asm -a
ASM home: /u01/app/121/grid
Password file: +DATA/orapwASM
ASM listener: LISTENER
ASM is enabled.
ASM instance count: 2
Cluster ASM listener: ASMNET1LSNR_ASM

Check clusterware resources 
[root@gract1 var]#  crs  | egrep 'asm|ASMNET|--|STATE_DETAILS'
Rescource NAME                 TARGET     STATE           SERVER       STATE_DETAILS                       
-------------------------      ---------- ----------      ------------ ------------------                  
ora.ASMNET1LSNR_ASM.lsnr       ONLINE     ONLINE          gract1       STABLE   
ora.ASMNET1LSNR_ASM.lsnr       ONLINE     ONLINE          gract2       STABLE   
ora.ASMNET1LSNR_ASM.lsnr       ONLINE     ONLINE          gract3       STABLE   

Resource NAME               INST   TARGET       STATE        SERVER          STATE_DETAILS
--------------------------- ----   ------------ ------------ --------------- -----------------------------------------
ora.asm                        1   ONLINE       ONLINE       gract1          Started,STABLE  
ora.asm                        3   ONLINE       ONLINE       gract2          Started,STABLE a 

ASM listener details 
[grid@gract2 ~]$ lsnrctl services ASMNET1LSNR_ASM 
LSNRCTL for Linux: Version 12.1.0.1.0 - Production on 07-AUG-2014 17:14:01
Copyright (c) 1991, 2013, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=ASMNET1LSNR_ASM)))
Services Summary...
Service "+ASM" has 2 instance(s).
  Instance "+ASM2", status READY, has 2 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.2.112)(PORT=1521)))
      "DEDICATED" established:1213 refused:0 state:ready
         LOCAL SERVER
  Instance "+ASM3", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.2.113)(PORT=1521)))
The command completed successfully

The ASM instance  automatically registers with ASMListener at port 1521 using the asm or cluster_interconnect,asm network
[grid@gract1 ASM]$  oifcfg getif -type cluster_interconnect,asm
eth2  192.168.2.0  global  cluster_interconnect,asm

--> Instance "+ASM2" runs on gract2  serviced  by ASMNET1LSNR_ASM  listening on 192.168.2.112 Port 1521
    Instance "+ASM3" runs on gract3  serviced  by ASMNET1LSNR_ASM  listening on 192.168.2.113 Port 1521

Check resource dependencies
[root@gract1 var]#  crsctl status resource ora.asm -f | grep -i DEP
RELOCATE_BY_DEPENDENCY=0
START_DEPENDENCIES=weak(ora.LISTENER.lsnr) pullup(ora.ASMNET1LSNR_ASM.lsnr) hard(ora.ASMNET1LSNR_ASM.lsnr)
STOP_DEPENDENCIES=hard(intermediate:ora.ASMNET1LSNR_ASM.lsnr)
--> ASM instance starts and stops ASM listener 

ADVM Proxy

  • ASM Dynamic Volume Manager (ADVM) Proxy is a special Oracle instance.
  • It enables ADVM to connect to Flex ASM and is required to run on the same node as ADVM and ACFS. 
  • It can be shutdown when ACFS is not running.
  • The ADVM proxy instance has its ORACLE_SID set to +APX<node number>
ADVM Proxy details
$ my_crs_stat
NAME                           TARGET     STATE           SERVER       STATE_DETAILS   
-------------------------      ---------- ----------      ------------ ------------------
                               Name       Target          State        Server State
ora.proxy_advm                 ONLINE     ONLINE          gract1       STABLE 
ora.proxy_advm                 ONLINE     ONLINE          gract2       STABLE 
ora.proxy_advm                 ONLINE     ONLINE          gract3       STABLE 

$ ps -elf | grep pmon | grep APX
0 S grid      4019     1  0  80   0 - 350587 semtim 07:53 ?       00:00:00 apx_pmon_+APX3

$ srvctl status asm -proxy
ADVM proxy is running on node gract3,gract2,gract1

Stop/Start ADVM proxy server on a certain node:
$ srvctl status asm -proxy
ADVM proxy is running on node gract3,gract2,gract1
$  srvctl stop asm -proxy -node gract3
$ srvctl status asm -proxy
ADVM proxy is running on node gract2,gract1
$  ps -elf | grep pmon | grep APX
$ srvctl start asm -proxy -node gract3
$  ps -elf | grep pmon | grep APX
0 S grid      4967     1  0  80   0 - 350587 semtim 08:09 ?       00:00:00 apx_pmon_+APX3
$ srvctl status asm -proxy
ADVM proxy is running on node gract3,gract2,gract1

 

Remove and add an ASM instance to  our FLEX ASM cluster

Remove an ASM instance to  our FLEX ASM cluster 
[grid@gract1 ~]$ srvctl modify asm -count 2
[grid@gract1 ~]$  crs | egrep 'STATE|asm|--'
NAME                           TARGET     STATE           SERVER       STATE_DETAILS   
-------------------------      ---------- ----------      ------------ ------------------
ora.asm                        ONLINE     ONLINE          gract3       STABLE 
ora.asm                        ONLINE     ONLINE          gract2       STABLE 

[grid@gract1 ~]$ srvctl status asm -detail
ASM is running on gract3,gract2
ASM is enabled.

Add an ASM instance to  our FLEX ASM cluster  
[grid@gract1 ~]$ srvctl modify asm -count 3
[grid@gract1 ~]$  srvctl status asm -detail
ASM is running on gract3,gract2
ASM is enabled.
--> Here we need to startup the ASM instance manually
[grid@gract1 ~]$ srvctl start asm
[grid@gract1 ~]$  crs | egrep 'STATE|asm|--'
NAME                           TARGET     STATE           SERVER       STATE_DETAILS   
-------------------------      ---------- ----------      ------------ ------------------
ora.asm                        ONLINE     ONLINE          gract1       STABLE 
ora.asm                        ONLINE     ONLINE          gract3       STABLE 
ora.asm                        ONLINE     ONLINE          gract2       STABLE 

[grid@gract1 ~]$  srvctl status asm -detail
ASM is running on gract3,gract2,gract1
ASM is enabled.

Some useful commands to administer Flex ASM

$ srvctl config asm
ASM home: /u01/app/121/grid
Password file: +DATA/orapwASM
ASM listener: LISTENER
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM

$  srvctl status asm -detail
ASM is running on gract3,gract2,gract1
ASM is enabled.

$  srvctl status asm -proxy -detail
ADVM proxy is running on node gract3,gract2,gract1
ADVM proxy is enabled

$ crsctl status resource ora.asm -f | grep CARDINALITY=
CARDINALITY=3

Monitor Flex ASM Connections:

SQL> select INST_ID,GROUP_NUMBER, INSTANCE_NAME, DB_NAME, INSTANCE_NAME||':'||DB_NAME client_id, STATUS from gv$asm_client;

   INST_ID GROUP_NUMBER INSTANCE_N DB_NAME  CLIENT_ID         STATUS
---------- ------------ ---------- -------- -------------------- ------------
     3          1 +ASM3       +ASM     +ASM3:+ASM         CONNECTED
     3          1 gract1_1   gract1   gract1_1:gract1     CONNECTED
     3          1 gract1_2   gract1   gract1_2:gract1     CONNECTED
     3          0 +APX1       +APX     +APX1:+APX         CONNECTED
     3          0 +APX2       +APX     +APX2:+APX         CONNECTED
     3          0 +APX3       +APX     +APX3:+APX         CONNECTED
     3          1 -MGMTDB    _mgmtdb  -MGMTDB:_mgmtdb     CONNECTED
     2          1 gract1_3   gract1   gract1_3:gract1     CONNECTED

Reduce/Increase ASM instance count

$  srvctl config asm
ASM home: /u01/app/121/grid
Password file: +DATA/orapwASM
ASM listener: LISTENER
ASM instance count: 2
Cluster ASM listener: ASMNET1LSNR_ASM

$  srvctl status asm -detail
ASM is running on gract3,gract2
$ srvctl modify asm -count 3  
$ srvctl start asm -n gract1
$ srvctl status asm -detail
ASM is running on gract3,gract2,gract1
ASM is enabled.

$  srvctl config asm
ASM home: /u01/app/121/grid
Password file: +DATA/orapwASM
ASM listener: LISTENER
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM

Relocate ASM  connections


Due to the BUG mentioned below I've tested ASM relocation feature against 12.1.0.2 
   Bug 18003355 : LNX64-12.1-ASM, EXECUTE "RELOCATE" COMMAND,HIT ORA-15344 for further details

Setup testbase by reducing ASM instances to 2 in our 3-node cluster running 3 HUBs
[oracle@gract1 ~]$ srvctl modify asm -count 2
[oracle@gract1 ~]$ srvctl status asm -detail
ASM is running on gract2,gract1
ASM is enabled.
--> ASM is running on gract1,gract2

SQL>  @asm_cli
INSTANCE_NAME
--------------------
+ASM1
   INST_ID GROUP_NUMBER INSTANCE_NAME         DB_NAME  CLIENT_ID        STATUS
---------- ------------ -------------------- -------- -------------------- ------------
     1          2 cdbn1             cdbn     cdbn1:cdbn       CONNECTED
     1          2 cdbn3             cdbn     cdbn3:cdbn       CONNECTED
     2          2 cdbn2             cdbn     cdbn2:cdbn       CONNECTED

  cdbn2:cdbn -> connected to instance id 2 -> local  ASM client 
  cdbn3:cdbn -> connected to instance id 1 -> Remote ASM client
  cdbn1:cdbn -> connected to instance id 1 -> local  ASM  client

What happens when we relocate an client ?  
1) Relocate client cdbn2:cdbn when connected to INST_ID 1 
SQL> alter system relocate client 'cdbn2:cdbn';
  alter system relocate client 'cdbn2:cdbn'
  *
  ERROR at line 1:  
  ORA-15344: client cdbn2:cdbn@+ASM1 not found
     INST_ID GROUP_NUMBER INSTANCE_NAME         DB_NAME  CLIENT_ID        STATUS
  ---------- ------------ -------------------- -------- -------------------- ------------
     1          2 cdbn1             cdbn     cdbn1:cdbn       CONNECTED
     1          2 cdbn3             cdbn     cdbn3:cdbn       CONNECTED
     2          2 cdbn2             cdbn     cdbn2:cdbn       CONNECTED
  --> Error ORA-15344 . 
      We need connect INST_ID 2 to relocate that client   'cdbn2:cdbn';

2) Relocate the local client cdbn1:cdbn on INST_ID 1
SQL> alter system relocate client 'cdbn1:cdbn';
  System altered.
+ASM1 alert.log  
  Mon Aug 11 14:19:10 2014
  NOTE: relocating client cdbn1:cdbn:gract (reg:3580848907)
  NOTE: cleaned up ASM client cdbn1:cdbn:gract connection state  
  Mon Aug 11 14:19:10 2014
  NOTE: Flex client id 0x10005 [cdbn1:cdbn:gract] attempting to reconnect
  NOTE: registered owner id 0x10005 for cdbn1:cdbn:gract (reconnecting)
  NOTE: Flex client cdbn1:cdbn:gract re-registered, osid 17384, mbr 0x0, asmb 32283 (reg:1445977979)
  NOTE: client [cdbn1:cdbn:gract] declared 0 additional pending writes

   INST_ID GROUP_NUMBER INSTANCE_NAME         DB_NAME  CLIENT_ID        STATUS
  ---------- ------------ -------------------- -------- -------------------- ------------
     1          2 cdbn1             cdbn     cdbn1:cdbn       CONNECTED
     1          2 cdbn3             cdbn     cdbn3:cdbn       CONNECTED
     2          2 cdbn2             cdbn     cdbn2:cdbn       CONNECTED
  --> ASM doesn't relocate the instance - instead it just recconnects to the local ASM instance 

3) Relocate the Remote ASM client connection
SQL> alter system relocate client 'cdbn3:cdbn';
ASM1+ alert.log 
  Mon Aug 11 14:25:47 2014
  NOTE: relocating client cdbn3:cdbn:gract (reg:959447971)
  NOTE: cleaned up ASM client cdbn3:cdbn:gract connection state
  Mon Aug 11 14:25:49 2014
  NOTE: detected orphaned client id 0x10003.
  NOTE: client cdbn3:cdbn:gract id 0x10003 has reconnected to ASM inst +ASM2 (reg:309839056), or has been fenced
   INST_ID GROUP_NUMBER INSTANCE_NAME         DB_NAME  CLIENT_ID        STATUS
  ---------- ------------ -------------------- -------- -------------------- ------------
     1          2 cdbn1             cdbn     cdbn1:cdbn       CONNECTED
     2          2 cdbn2             cdbn     cdbn2:cdbn       CONNECTED
     2          2 cdbn3             cdbn     cdbn3:cdbn       CONNECTED
  Note : The relocate not always worked ( sometimes I got a reconnect even for as REMOTE asn client )
         In that case just repeat the the relocate command 

  To relocate ASM instance back to INST_ID 1 connect to +ASM2 instance and run 
  SQL> alter system relocate client 'cdbn3:cdbn';

Summary: Local  ASM clients always tries to re-connect locally 
         Remote ASM clients really fail over the remote note 

Force a Flex ASM network connection by reducing ASM instance count

$  srvctl status asm -detail
ASM is running on gract3,gract2,gract1
ASM is enabled.
$ srvctl modify asm -count 2
$  srvctl status asm -detail
ASM is running on gract3,gract2
ASM is enabled.
--> ASM instance dropped on gract1 - Connected session still working
SQL> select INSTANCE_NUMBER,INSTANCE_NAME,STATUS, HOST_NAME    from v$instance;
INSTANCE_NUMBER INSTANCE_N STATUS    HOST_NAME
--------------- ---------- ------------ ------------------------------
          1 gract1_1   OPEN     gract1.example.com
--> alert.log on  gract1 reports that ASM is reconnecting to ASM instance +ASM3
Thu Oct 03 10:35:35 2013
NOTE: ASMB registering with ASM instance as client 0x10004 (reg:2698621261)
NOTE: ASMB connected to ASM instance +ASM3 (Flex mode; client id 0x10004)
NOTE: ASMB rebuilding ASM server state
NOTE: ASMB rebuilt 1 (of 1) groups
NOTE: ASMB rebuilt 17 (of 17) allocated files
NOTE: fetching new locked extents from server
NOTE: 0 locks established; 0 pending writes sent to server
SUCCESS: ASMB reconnected & completed ASM server state
SQL> select INST_ID,GROUP_NUMBER, INSTANCE_NAME, DB_NAME, INSTANCE_NAME||':'||DB_NAME client_id, STATUS from gv$asm_client
      where DB_NAME = 'gract1';
   INST_ID GROUP_NUMBER INSTANCE_N DB_NAME  CLIENT_ID         STATUS
---------- ------------ ---------- -------- -------------------- ------------
     3          1 gract1_1   gract1   gract1_1:gract1     CONNECTED  <-- ASM connection connected via Network
     3          1 gract1_2   gract1   gract1_2:gract1     CONNECTED
     2          1 gract1_3   gract1   gract1_3:gract1     CONNECTED

 

Force a Flex ASM network connection by killing  ASM asmb process

Testing ASM instance crash 

Prepare a db session as user scott connecting to PDB1 using CDBN1
[oracle@gract1 ~]$ sqlplus scott/tiger@gract1/PDB1
SQL> select INSTANCE_NUMBER,INSTANCE_NAME,STATUS, HOST_NAME    from v$instance;
INSTANCE_NUMBER INSTANCE_N STATUS    HOST_NAME
--------------- ---------- ------------ ------------------------------
          1 cdbn1       OPEN     gract1.example.com
SQL> 
SQL>  show con_name
CON_NAME
------------------------------
PDB1
SQL> select user from dual;
USER
------------------------------
SCOTT
--> Don't close this session we will test whether this connection survives the ASM instance crash

Connect to the local +ASM1 instance  and check ASM client status 
SQL> @asm_cli
INSTANCE_NAME
--------------------
+ASM1
   INST_ID GROUP_NUMBER INSTANCE_NAME         DB_NAME  CLIENT_ID        STATUS
---------- ------------ -------------------- -------- -------------------- ------------
     1          2 cdbn1             cdbn     cdbn1:cdbn       CONNECTED
     1          2 cdbn3             cdbn     cdbn3:cdbn       CONNECTED
     2          2 cdbn2             cdbn     cdbn2:cdbn       CONNECTED
+ASM1 instance serves a local client cdbn1:cdbn and a remote client :  cdbn3:cdbn
Lets kill +ASM1 instance 
[grid@gract1 trace]$ ps -elf | grep -i asm_asmb
0 S grid     32119     1  0  80   0 - 361252 pipe_w 09:37 ?       00:00:00 asm_asmb_+ASM1
[root@gract1 var]#  kill -9  32119

Connect to +ASM2 instance and check ASM status 
INSTANCE_NAME
--------------------
+ASM2
   INST_ID GROUP_NUMBER INSTANCE_NAME         DB_NAME  CLIENT_ID        STATUS
---------- ------------ -------------------- -------- -------------------- ------------
     2          2 cdbn1             cdbn     cdbn1:cdbn       CONNECTED
     2          2 cdbn2             cdbn     cdbn2:cdbn       CONNECTED
     2          2 cdbn3             cdbn     cdbn3:cdbn       CONNECTED
--> Now we have 2 REMOTE ASM clients cdbn1:cdbn and cdbn3:cdbn
    All ASM operations are now serviced by INST_ID 2 ( +ASM2 )

Check SQL session status 
SQL>  select user from dual;
USER
------------------------------
SCOTT
--> SQL session survived the ASM failover 

Note ASM instance is automatically restarted due to CW resources monitoring
Check CW resources  
[root@gract1 var]#  crs  | egrep 'asm|---|INST'
-------------------------      ---------- ----------      ------------ ------------------                  
Resource NAME               INST   TARGET       STATE        SERVER          STATE_DETAILS
--------------------------- ----   ------------ ------------ --------------- -----------------------------------------
ora.asm                        1   ONLINE       ONLINE       gract1          Started,STABLE  
ora.asm                        3   ONLINE       ONLINE       gract2          Started,STABLE  

Relocate ASM clients as needed
SQL> alter system relocate client 'cdbn3:cdbn';
System altered.
SQL>   alter system relocate client 'cdbn1:cdbn';
System altered.
SQL> @asm_cli
INSTANCE_NAME
--------------------
+ASM2
   INST_ID GROUP_NUMBER INSTANCE_NAME         DB_NAME  CLIENT_ID        STATUS
---------- ------------ -------------------- -------- -------------------- ------------
     1          2 cdbn1             cdbn     cdbn1:cdbn       CONNECTED
     1          2 cdbn3             cdbn     cdbn3:cdbn       CONNECTED
     2          2 cdbn2             cdbn     cdbn2:cdbn       CONNECTED
Reference

3 thoughts on “12c feature: FLEX ASM”

  1. Hi Helmut,

    Is the output below, a correct output for a two node 12.1.0.2 RAC one node cluster installation with FLEX ASM option?

    ora.asm
    1 ONLINE ONLINE racnode1 STABLE
    2 ONLINE ONLINE racnode2 Started,STABLE
    3 OFFLINE OFFLINE STABLE

    I wonder because of the difference in the status “Started,STABLE” and “STABLE” of the two nodes. I have an example of a three node RAC cluster (not RAC one node) where all three entries are in “Started,STABLE” status.

    Best regards,
    Evagelos

Leave a Reply

Your email address will not be published. Required fields are marked *