12c feature : What-If Command Evaluation with serverpool

Current Cluster Configuration

ASM Configuration
[root@gract3 ~]#   asmcmd showclustermode
ASM cluster : Flex mode enabled

[root@gract3 ~]#  srvctl status asm -detail
ASM is running on gract2,gract1
ASM is enabled.

[root@gract3 ~]# crsctl status resource ora.asm -f | grep CARDINALITY=
CARDINALITY=2

RAC configuration
[root@gract3 ~]# olsnodes -c
gract
[root@gract3 ~]#   crsctl get node role status -all
Node 'gract1' active role is 'hub'
Node 'gract2' active role is 'hub'
Node 'gract3' active role is 'hub'
Summary : - This cluster is running 3 HUB nodes where gract2 and gract1 are currently hosting our 2 ASM FLEX instances 

Following CRS resource output reflects the current status  
[root@gract1 Desktop]# crs
*****  Local Resources: *****
Rescource NAME                 TARGET     STATE           SERVER       STATE_DETAILS                       
-------------------------      ---------- ----------      ------------ ------------------                  
ora.ACFS_DG1.ACFS_VOL1.advm    ONLINE     ONLINE          gract1       Volume device /dev/a sm/acfs_vol1-443 isonline,STABLE
ora.ACFS_DG1.ACFS_VOL1.advm    ONLINE     ONLINE          gract2       Volume device /dev/a sm/acfs_vol1-443 isonline,STABLE
ora.ACFS_DG1.ACFS_VOL1.advm    ONLINE     OFFLINE         gract3       Unable to connect toASM,STABLE
ora.ACFS_DG1.dg                ONLINE     ONLINE          gract1       STABLE   
ora.ACFS_DG1.dg                ONLINE     ONLINE          gract2       STABLE   
ora.ACFS_DG1.dg                ONLINE     OFFLINE         gract3       STABLE   
ora.ASMNET1LSNR_ASM.lsnr       ONLINE     ONLINE          gract1       STABLE   
ora.ASMNET1LSNR_ASM.lsnr       ONLINE     ONLINE          gract2       STABLE   
ora.ASMNET1LSNR_ASM.lsnr       ONLINE     OFFLINE         gract3       STABLE   
ora.DATA.dg                    ONLINE     ONLINE          gract1       STABLE   
ora.DATA.dg                    ONLINE     ONLINE          gract2       STABLE   
ora.DATA.dg                    ONLINE     OFFLINE         gract3       STABLE   
ora.LISTENER.lsnr              ONLINE     ONLINE          gract1       STABLE   
ora.LISTENER.lsnr              ONLINE     ONLINE          gract2       STABLE   
ora.LISTENER.lsnr              ONLINE     OFFLINE         gract3       STABLE   
ora.acfs_dg1.acfs_vol1.acfs    ONLINE     ONLINE          gract1       mounted on /u01/acfs /acfs-vol1,STABLE
ora.acfs_dg1.acfs_vol1.acfs    ONLINE     ONLINE          gract2       mounted on /u01/acfs /acfs-vol1,STABLE
ora.acfs_dg1.acfs_vol1.acfs    ONLINE     OFFLINE         gract3       (1) volume /u01/acfs /acfs-vol1 offline,STABLE
ora.net1.network               ONLINE     ONLINE          gract1       STABLE   
ora.net1.network               ONLINE     ONLINE          gract2       STABLE   
ora.net1.network               ONLINE     ONLINE          gract3       STABLE   
ora.ons                        ONLINE     ONLINE          gract1       STABLE   
ora.ons                        ONLINE     ONLINE          gract2       STABLE   
ora.ons                        ONLINE     OFFLINE         gract3       STABLE   
ora.proxy_advm                 ONLINE     ONLINE          gract1       STABLE   
ora.proxy_advm                 ONLINE     ONLINE          gract2       STABLE   
ora.proxy_advm                 ONLINE     OFFLINE         gract3       STABLE   
*****  Cluster Resources: *****
Resource NAME               INST   TARGET       STATE        SERVER          STATE_DETAILS
--------------------------- ----   ------------ ------------ --------------- -----------------------------------------
MyTestVIP                      1   ONLINE       OFFLINE      -               STABLE  
My_apache                      1   ONLINE       OFFLINE      -               STABLE  
ora.LISTENER_SCAN1.lsnr        1   ONLINE       ONLINE       gract1          STABLE  
ora.LISTENER_SCAN2.lsnr        1   ONLINE       ONLINE       gract2          STABLE  
ora.LISTENER_SCAN3.lsnr        1   ONLINE       ONLINE       gract1          STABLE  
ora.MGMTLSNR                   1   ONLINE       ONLINE       gract1          169.254.235.72 192.1 68.2.111,STABLE
ora.asm                        1   ONLINE       ONLINE       gract1          STABLE  
ora.asm                        3   ONLINE       ONLINE       gract2          STABLE  
ora.cvu                        1   ONLINE       ONLINE       gract1          STABLE  
ora.dw.db                      1   ONLINE       ONLINE       gract2          Open,STABLE  
ora.dw.db                      2   ONLINE       OFFLINE      -               STABLE  
ora.erp.db                     1   ONLINE       ONLINE       gract3          Open,STABLE  
ora.erp.db                     2   OFFLINE      OFFLINE      -               Instance Shutdown,ST ABLE
ora.erp.db                     3   ONLINE       ONLINE       gract1          Open,STABLE  
ora.gns                        1   ONLINE       ONLINE       gract1          STABLE  
ora.gns.vip                    1   ONLINE       ONLINE       gract1          STABLE  
ora.gract1.vip                 1   ONLINE       ONLINE       gract1          STABLE  
ora.gract2.vip                 1   ONLINE       ONLINE       gract2          STABLE  
ora.gract3.vip                 1   ONLINE       INTERMEDIATE gract2          FAILED OVER,STABLE 
ora.mgmtdb                     1   ONLINE       ONLINE       gract1          Open,STABLE  
ora.oc4j                       1   ONLINE       ONLINE       gract1          STABLE  
ora.scan1.vip                  1   ONLINE       ONLINE       gract1          STABLE  
ora.scan2.vip                  1   ONLINE       ONLINE       gract2          STABLE  
ora.scan3.vip                  1   ONLINE       ONLINE       gract1          STABLE  
Following instances 
------ RAC Database Info ---------
Database name: dw Server pools: STANDARD_PRIORITY 
Instance dw_1 is running on node gract2

Database name: ERP Server pools: TOP_PRIORITY 
Instance ERP_1 is running on node gract3
Instance ERP_3 is running on node gract1
------ Serverpool Info Config ---------
NAME=Free                          IMPORTANCE=0        MIN_SIZE=0          MAX_SIZE=-1         SERVER_CATEGORY=                    
NAME=ora.TOP_PRIORITY              IMPORTANCE=5        MIN_SIZE=1          MAX_SIZE=2          SERVER_CATEGORY=ora.hub.category    
NAME=ora.STANDARD_PRIORITY         IMPORTANCE=5        MIN_SIZE=1          MAX_SIZE=2          SERVER_CATEGORY=ora.hub.category    

------ Serverpool Info ---------
Server pool name: Free             Active servers count: 0            Active server names:                 
Server pool name: TOP_PRIORITY     Active servers count: 2            Active server names: gract1,gract3  
           NAME=gract1 STATE=ONLINE           NAME=gract3 STATE=ONLINE 
Server pool name: STANDARD_PRIORITYActive servers count: 1            Active server names: gract2         
           NAME=gract2 STATE=ONLINE

What Rescource Changes can we expect when server crashes or a CRS stack shutdown occurs

Run crsctl eval to get a rought idea what is going on
[root@gract3 ~]#   crsctl eval delete server gract2

-> Server 'gract2' will be removed from pools [ora.STANDARD_PRIORITY]     As crs is down gract2 need to leave the pool
Server 'gract3' will be moved from pools      [ora.TOP_PRIORITY] to       gract2 needs to move to STANDARD_PRIORITY
                                           pools [ora.STANDARD_PRIORITY]  pool to satisfiy  MIN_SIZE=1 pool settings
Resource 'ora.DATA.dg' (gract2) will be in state    [OFFLINE]             As our FLEX ASM runs on gract2 DG are going 
                                                                          OFFLINE
Resource 'ora.asm' (3/1) will be in state           [OFFLINE]             After DGs are OFFLINE we can stop the ASM 
                                                                          server
Resource 'ora.dw.db' (1/1) will be in state         [OFFLINE]             DW instance needs to go OFFLINE as server 
                                                                          gract2 leaves pool STANDARD_PRIORITY

Resource 'ora.erp.db' (1/1) will be in state        [OFFLINE]             ERP_1 needs to set OFFLINE as server gract3 leaves that pool

Resource 'ora.DATA.dg' (gract3) will be in state    [ONLINE|INTERMEDIATE] Flex ASM restarted on gract3
                                                    on server [gract3]

It seems no all events are displayed - but it is a good test to get a rough idea what is going on.
The startup of DW instance was missing for  crsctl eval delete server gract2 command.

Afer stopping CRS  on server gract2 by runinng 
[root@gract2 ~]#   crsctl stop crs

The following changes are active now 
------ RAC Database Info ---------
Database name: dw Server pools: STANDARD_PRIORITY
Instance dw_2 is running on node gract3

Database name: ERP Server pools: TOP_PRIORITY
Instance ERP_3 is running on node gract1

[grid@gract1 ~/PM]$  srvctl status asm -detail
ASM is running on gract3,gract1
ASM is enabled.

Summary

  • crsctl eval is a good tool to verify clusterware actions happening  for a certain command
  • Due to serverpool operations even instances not running on affected server can go OFFLINE
  • Use  crsctl eval delete server .. to track changes for a CRS stack outage

Full output from running: crsctl eval delete server ..

[root@gract2 ~]# crsctl eval delete server gract3

Stage Group 1:
--------------------------------------------------------------------------------
Stage Number    Required    Action    
--------------------------------------------------------------------------------

     1            Y        Server 'gract2' will be moved from pools
                [ora.TOP_PRIORITY] to pools
                [ora.STANDARD_PRIORITY]
            Y        Server 'gract3' will be removed from pools
                [ora.STANDARD_PRIORITY]
            Y        Resource 'ora.ACFS_DG1.ACFS_VOL1.advm' (gract3)
                will be in state [OFFLINE]
            Y        Resource 'ora.ACFS_DG1.dg' (gract3) will be in
                state [OFFLINE]
            Y        Resource 'ora.ASMNET1LSNR_ASM.lsnr' (gract3)
                will be in state [OFFLINE]
            Y        Resource 'ora.DATA.dg' (gract3) will be in state
                [OFFLINE]
            Y        Resource 'ora.LISTENER.lsnr' (gract3) will be in
                state [OFFLINE]
            Y        Resource 'ora.LISTENER_SCAN1.lsnr' (1/1) will be
                in state [OFFLINE]
            Y        Resource 'ora.acfs_dg1.acfs_vol1.acfs' (gract3)
                will be in state [OFFLINE]
            Y        Resource 'ora.asm' (3/1) will be in state
                [OFFLINE]
            Y        Resource 'ora.dw.db' (1/1) will be in state
                [OFFLINE]
            Y        Resource 'ora.gract3.vip' (1/1) will be in state
                [OFFLINE]
            Y        Resource 'ora.net1.network' (gract3) will be in
                state [OFFLINE]
            Y        Resource 'ora.ons' (gract3) will be in state
                [OFFLINE]
            Y        Resource 'ora.proxy_advm' (gract3) will be in
                state [OFFLINE]
            Y        Resource 'ora.scan1.vip' (1/1) will be in state
                [OFFLINE]

     2            N        Create new group (Stage Group = 2)
            Y        Resource 'ora.asm' (3/1) will be in state
                [ONLINE|INTERMEDIATE] on server [gract2]
            Y        Resource 'ora.gract3.vip' (1/1) will be in state
                [ONLINE|INTERMEDIATE] on server [gract2]
            Y        Resource 'ora.scan1.vip' (1/1) will be in state
                [ONLINE] on server [gract1]

     3            Y        Resource 'ora.LISTENER_SCAN1.lsnr' (1/1) will be
                in state [ONLINE|INTERMEDIATE] on server
                [gract1]
            Y        Resource 'ora.erp.db' (1/1) will be in state
                [OFFLINE]

--------------------------------------------------------------------------------

Stage Group 2:
--------------------------------------------------------------------------------
Stage Number    Required    Action    
--------------------------------------------------------------------------------

     1            N        Resource 'ora.DATA.dg' (gract2) will be in state
                [ONLINE|INTERMEDIATE] on server [gract2]

--------------------------------------------------------------------------------

crsctl commands supporting eval switch

  crsctl eval start resource {<resname>|-w <filter>}[-n server] [-f]
  crsctl eval stop resource {<resname>|-w <filter>} [-f]
  crsctl eval relocate resource {<resName> | {<resName>|-all} -s <server> | -w <filter>} {-n <server>} [-f]
  crsctl eval add serverpool <spName> [-file <filePath> | -attr  "<attrName>=<value>[,...]"] [-f] [-admin [-l <level>] [-x] [-a]]
  crsctl eval modify serverpool <spName> {-file <filePath> | -attr  "<attrName>=<value>[,...]"} [-f] [-admin [-l <level>] [-x] [-a]]
  crsctl eval delete serverpool <spName>  [-admin [-l <level>] [-x] [-a]]
  crsctl eval add server <serverName> [-file <filePath> | -attr  "<attrName>=<value>[,...]"] [-f] [-admin [-l <level>] [-x] [-a]]
  crsctl eval relocate server <serverName> -to <toPool> [-f] [-admin [-l <level>] [-x] [-a]]
  crsctl eval delete server <serverName> [-f] [-admin [-l <level>] [-x] [-a]]
  crsctl eval add resource <resName> -type <typeName> [-file <filePath> | -attr  "<attrName>=<value>[,...]"]  [-f]
  crsctl eval modify resource <resName> -attr "<attrName>=<value>[,...]" [-f] 
  crsctl eval fail resource {<resname> | -w <filter>} [-n <server>]
  crsctl eval activate policy <policyName> [-f] [-admin [-l <level>] [-x] [-a]]

Reference

 

3 thoughts on “12c feature : What-If Command Evaluation with serverpool”

Leave a Reply

Your email address will not be published. Required fields are marked *