Managing Resources

Overview

RESOURCE_TYPE can be database, instance, service, listener etc
RESOURCE_NAME is the name from crs_stat command for pre-11.2, or the name from "crsctl stat res" for 11gR2, i.e. ora.racdb.db
RESOURCE_HOME refers to the ORACLE_HOME that the resource runs off, i.e. RDBMS_HOME for database, instance and service resource;
RESOURCE_OWNER refers to the OS user that owns the RESOURCE_HOME, i.e. grid user for GRID_HOME.
B. Resource must be managed by srvctl from RESOURCE_HOME/bin

Resource must be managed by srvctl from $RESOURCE_HOME where resource is running off, for example, vip from $GRID_HOME or 
$CRS_HOME, 11.2 .db in 11.2 RDBMS_HOME, and 11.1 .db in 11.1 RDBMS_HOME etc.

If wrong srvctl is use an error will be reported: 
PRCD-1229 : An attempt to access configuration of database cdb was rejected because its version 
12.1.0.1.0 differs from the program version 12.1.0.2.0. 
Instead run the program from /u01/app/oracle/product/121/racdb.
  • How to Delete From or Add Resource to OCR in Oracle Clusterware (Doc ID 1069369.1)

Display local and clusterwide resources

Local resources ( based on crsctl stat res -t -init ) 
# my_crs_stat_init
NAME                           TARGET     STATE           SERVER       STATE_DETAILS   
-------------------------      ---------- ----------      ------------ ------------------
ora.asm                        ONLINE     ONLINE          grac41       Started 
ora.cluster_interconnect.haip  ONLINE     ONLINE          grac41        
ora.crf                        ONLINE     ONLINE          grac41        
ora.crsd                       ONLINE     ONLINE          grac41        
ora.cssd                       ONLINE     ONLINE          grac41        
ora.cssdmonitor                ONLINE     ONLINE          grac41        
ora.ctssd                      ONLINE     ONLINE          grac41       OBSERVER 
ora.diskmon                    OFFLINE    OFFLINE                       
ora.drivers.acfs               ONLINE     ONLINE          grac41        
ora.evmd                       ONLINE     ONLINE          grac41        
ora.gipcd                      ONLINE     ONLINE          grac41        
ora.gpnpd                      ONLINE     ONLINE          grac41        
ora.mdnsd                      ONLINE     ONLINE          grac41    
Note : ora.diskmon is OFFLINE for Non-Exadata system . This is the expected behavior. For details read
11.2.0.3 Grid Infrastructure diskmon Will be Offline by Default in Non-Exadata Environment (Doc ID 1346881.1)

Stop and start a local resources
# crsctl stop resource  ora.crf -init
# crsctl start resource  ora.crf -init
Note you should start an stop HAIP with an active RAC 
# crsctl stop  resource  ora.cluster_interconnect.haip  -init
# crsctl start  resource  ora.cluster_interconnect.haip  -init

Display CLusterwide resources  (based on crsctl stat res -t )
# my_crs_stat
NAME                           TARGET     STATE           SERVER       STATE_DETAILS   
-------------------------      ---------- ----------      ------------ ------------------
ora.DATA.dg                    ONLINE     ONLINE          grac41        
ora.DATA.dg                    OFFLINE    ONLINE          grac42        
ora.DATA.dg                    OFFLINE    ONLINE          grac43        
ora.FRA.dg                     ONLINE     ONLINE          grac41        
ora.FRA.dg                     OFFLINE    ONLINE          grac42        
ora.FRA.dg                     OFFLINE    ONLINE          grac43        
ora.LISTENER.lsnr              ONLINE     ONLINE          grac41        
ora.LISTENER.lsnr              ONLINE     ONLINE          grac42        
ora.LISTENER.lsnr              ONLINE     ONLINE          grac43        
ora.OCR.dg                     ONLINE     ONLINE          grac41        
ora.OCR.dg                     ONLINE     ONLINE          grac42        
ora.OCR.dg                     ONLINE     ONLINE          grac43        
ora.SSD.dg                     ONLINE     ONLINE          grac41        
ora.SSD.dg                     ONLINE     ONLINE          grac42        
ora.SSD.dg                     ONLINE     ONLINE          grac43        
ora.asm                        ONLINE     ONLINE          grac41       Started 
ora.asm                        ONLINE     ONLINE          grac42       Started 
ora.asm                        ONLINE     ONLINE          grac43       Started 
ora.gsd                        OFFLINE    OFFLINE         grac41        
ora.gsd                        OFFLINE    OFFLINE         grac42        
ora.gsd                        OFFLINE    OFFLINE         grac43        
ora.net1.network               ONLINE     ONLINE          grac41        
ora.net1.network               ONLINE     ONLINE          grac42        
ora.net1.network               ONLINE     ONLINE          grac43        
ora.ons                        ONLINE     ONLINE          grac41        
ora.ons                        ONLINE     ONLINE          grac42        
ora.ons                        ONLINE     ONLINE          grac43        
ora.registry.acfs              ONLINE     ONLINE          grac41        
ora.registry.acfs              ONLINE     ONLINE          grac42        
ora.registry.acfs              ONLINE     ONLINE          grac43        
ora.LISTENER_SCAN1.lsnr        ONLINE     ONLINE          grac43        
ora.LISTENER_SCAN2.lsnr        ONLINE     ONLINE          grac41        
ora.LISTENER_SCAN3.lsnr        ONLINE     ONLINE          grac42        
ora.cvu                        ONLINE     ONLINE          grac42        
ora.gns                        ONLINE     ONLINE          grac43        
ora.gns.vip                    ONLINE     ONLINE          grac43        
ora.grac4.db                   ONLINE     ONLINE          grac41       Open 
ora.grac4.db                   ONLINE     ONLINE          grac42       Open 
ora.grac4.db                   ONLINE     ONLINE          grac43       Open 
ora.grac4.grac41.svc           ONLINE     ONLINE          grac41        
ora.grac4.grac42.svc           ONLINE     ONLINE          grac42        
ora.grac4.grac43.svc           ONLINE     ONLINE          grac43        
ora.grac4.report.svc           ONLINE     ONLINE          grac42        
ora.grac41.vip                 ONLINE     ONLINE          grac41        
ora.grac42.vip                 ONLINE     ONLINE          grac42        
ora.grac43.vip                 ONLINE     ONLINE          grac43        
ora.oc4j                       ONLINE     ONLINE          grac42        
ora.scan1.vip                  ONLINE     ONLINE          grac43        
ora.scan2.vip                  ONLINE     ONLINE          grac41        
ora.scan3.vip                  ONLINE     ONLINE          grac42        
procwatcher                    OFFLINE    OFFLINE                       
procwatcher                    ONLINE     ONLINE          grac43        
procwatcher                    ONLINE     ONLINE          grac42 

Display Resource Details
crsctl status resource  switches:
   -f display full configuration  
   -p dislpay static configuration 
   -v display runtime configuration 

Display all resources with full details  
#  crsctl status resource -f

Display a single resource with full details  
#  crsctl status resource ora.grac4.db -f
NAME=ora.grac4.db
TYPE=ora.database.type
ACL=owner:oracle:rwx,pgrp:oinstall:r--,other::r--,group:dba:r-x,group:oper:r-x,user:grid:r-x
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=
ACTIVE_PLACEMENT=1
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
AUTO_START=restore
CARDINALITY=3
CHECK_INTERVAL=1
CHECK_TIMEOUT=30
CLUSTER_DATABASE=true
DATABASE_TYPE=RAC
DB_UNIQUE_NAME=grac4
DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=database) PROPERTY(DB_UNIQUE_NAME= CONCAT(PARSE(%NAME%, ., 2), %USR_ORA_DOMAIN%, .)) ELEMENT(INSTANCE_NAME= %GEN_USR_ORA_INST_NAME%) ELEMENT(DATABASE_TYPE= %DATABASE_TYPE%)
DEGREE=1
DESCRIPTION=Oracle Database resource
ENABLED=1
FAILOVER_DELAY=0
FAILURE_INTERVAL=60
FAILURE_THRESHOLD=1
GEN_AUDIT_FILE_DEST=/u01/app/oracle/admin/grac4/adump
GEN_START_OPTIONS=
GEN_START_OPTIONS@SERVERNAME(grac41)=open
GEN_START_OPTIONS@SERVERNAME(grac42)=open
GEN_START_OPTIONS@SERVERNAME(grac43)=open
GEN_USR_ORA_INST_NAME=
GEN_USR_ORA_INST_NAME@SERVERNAME(grac41)=grac41
GEN_USR_ORA_INST_NAME@SERVERNAME(grac42)=grac42
GEN_USR_ORA_INST_NAME@SERVERNAME(grac43)=grac43
HOSTING_MEMBERS=
INSTANCE_FAILOVER=0
LOAD=1
LOGGING_LEVEL=1
MANAGEMENT_POLICY=AUTOMATIC
NLS_LANG=
NOT_RESTARTING_TEMPLATE=
OFFLINE_CHECK_INTERVAL=0
ONLINE_RELOCATION_TIMEOUT=0
ORACLE_HOME=/u01/app/oracle/product/11204/racdb
ORACLE_HOME_OLD=
PLACEMENT=restricted
PROFILE_CHANGE_TEMPLATE=
RESTART_ATTEMPTS=2
ROLE=PRIMARY
SCRIPT_TIMEOUT=60
SERVER_POOLS=ora.grac4
SPFILE=+DATA/grac4/spfilegrac4.ora
START_DEPENDENCIES=hard(ora.DATA.dg,ora.SSD.dg) weak(type:ora.listener.type,global:type:ora.scan_listener.type,uniform:ora.ons,global:ora.gns,ora.FRA.dg) pullup(ora.DATA.dg,ora.SSD.dg)
START_TIMEOUT=600
STATE_CHANGE_TEMPLATE=
STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.DATA.dg,shutdown:ora.SSD.dg)
STOP_TIMEOUT=600
TYPE_VERSION=3.2
UPTIME_THRESHOLD=1h
USR_ORA_DB_NAME=grac4
USR_ORA_DOMAIN=
USR_ORA_ENV=
USR_ORA_FLAGS=
USR_ORA_INST_NAME=
USR_ORA_INST_NAME@SERVERNAME(grac41)=grac41
USR_ORA_INST_NAME@SERVERNAME(grac42)=grac42
USR_ORA_INST_NAME@SERVERNAME(grac43)=grac43
USR_ORA_OPEN_MODE=open
USR_ORA_OPI=false
USR_ORA_STOP_MODE=immediate
VERSION=11.2.0.4.0

Sequence of stopping  all cluster  resources for a certain Node

Display all resources
[root@grac41 Desktop]# my_crs_stat | egrep 'NAME|grac41|---'
NAME                           TARGET     STATE           SERVER       STATE_DETAILS   
-------------------------      ---------- ----------      ------------ ------------------
ora.DATA.dg                    ONLINE     ONLINE          grac41        
ora.FRA.dg                     ONLINE     ONLINE          grac41        
ora.LISTENER.lsnr              ONLINE     ONLINE          grac41        
ora.OCR.dg                     ONLINE     ONLINE          grac41        
ora.SSD.dg                     ONLINE     ONLINE          grac41        
ora.asm                        ONLINE     ONLINE          grac41       Started 
ora.gsd                        OFFLINE    OFFLINE         grac41        
ora.net1.network               ONLINE     ONLINE          grac41        
ora.ons                        ONLINE     ONLINE          grac41        
ora.registry.acfs              ONLINE     ONLINE          grac41        
ora.LISTENER_SCAN2.lsnr        ONLINE     ONLINE          grac41        
ora.grac4.db                   ONLINE     ONLINE          grac41       Open 
ora.grac4.grac41.svc           ONLINE     ONLINE          grac41        
ora.grac41.vip                 ONLINE     ONLINE          grac41        
ora.scan2.vip                  ONLINE     ONLINE          grac41        
procwatcher                    ONLINE     ONLINE          grac41     

Stop listner, services and databases resources
# crsctl stop resource procwatcher -n grac41
# crsctl stop resource ora.LISTENER.lsnr   -n grac41
# crsctl stop resource ora.LISTENER_SCAN2.lsnr
# crsctl stop resource ora.grac4.grac41.svc
# crsctl stop resource ora.grac41.vip
# crsctl stop resource ora.grac4.db  -n grac41
# crsctl stop resource ora.registry.acfs  -n grac41
# crsctl stop resource ora.scan2.vip 

Stop all ASM Data groups and ASM resources
# crsctl stop resource  ora.DATA.dg -n grac41
# crsctl stop resource ora.FRA.dg
# crsctl stop resource ora.OCR.dg -n grac41
# crsctl stop resource ora.SSD.dg  -n grac41
# crsctl stop resource ora.asm   -n grac41

Stop Network resources
#  crsctl stop  resource  ora.ons -n grac41
#  crsctl stop  resource  ora.net1.network  -n grac41

Stop local resources 
$ crsctl stop resource ora.mdnsd -init
$ crsctl stop resource  ora.crsd -init
$ crsctl stop resource ora.ctssd  -init
$ crsctl stop resource ora.cluster_interconnect.haip -init
$ crsctl stop resource ora.evmd -init
$ crsctl stop resource ora.cssd  -init
$ crsctl stop resource ora.drivers.acfs -init
$ crsctl stop resource  ora.gipcd -init
$ crsctl stop resource  ora.gpnpd -init
[grid@grac41 ~]$  my_crs_stat_init
NAME                           TARGET     STATE           SERVER       STATE_DETAILS   
-------------------------      ---------- ----------      ------------ ------------------
ora.asm                        OFFLINE    OFFLINE                      Instance Shutdown
ora.cluster_interconnect.haip  OFFLINE    OFFLINE                       
ora.crf                        OFFLINE    OFFLINE                       
ora.crsd                       OFFLINE    OFFLINE                       
ora.cssd                       OFFLINE    OFFLINE                       
ora.cssdmonitor                OFFLINE    OFFLINE                       
ora.ctssd                      OFFLINE    OFFLINE                       
ora.diskmon                    OFFLINE    OFFLINE                       
ora.drivers.acfs               OFFLINE    OFFLINE                       
ora.evmd                       OFFLINE    OFFLINE                       
ora.gipcd                      OFFLINE    OFFLINE                       
ora.gpnpd                      OFFLINE    OFFLINE   

Start local resouces ( single step )
# crsctl start crs

Start ASM ( DG should be become available automtically
# crsctl start resource ora.asm   -n grac41a
 --> ora.grac41.vip gets started
 --> ora.LISTENER.lsnr gets started
 --> ora.registry.acfs gets started

Start Instance 
# crsctl start resource ora.grac4.grac41.svc
 --> ora.grac4.grac41.svc starts ora.scan2.vip
     ora.grac4.grac41.svc starts ora.LISTENER_SCAN2.lsnr
     ora.grac4.grac41.svc starts ora.grac4.db

$ my_crs_stat | grep grac41
ora.DATA.dg                    ONLINE     ONLINE          grac41        
ora.FRA.dg                     ONLINE     ONLINE          grac41        
ora.LISTENER.lsnr              ONLINE     ONLINE          grac41        
ora.OCR.dg                     ONLINE     ONLINE          grac41        
ora.SSD.dg                     ONLINE     ONLINE          grac41        
ora.asm                        ONLINE     ONLINE          grac41       Started 
ora.gsd                        OFFLINE    OFFLINE         grac41        
ora.net1.network               ONLINE     ONLINE          grac41        
ora.ons                        ONLINE     ONLINE          grac41        
ora.registry.acfs              ONLINE     ONLINE          grac41        
ora.LISTENER_SCAN2.lsnr        ONLINE     ONLINE          grac41        
ora.grac4.db                   ONLINE     ONLINE          grac41       Open 
ora.grac4.grac41.svc           ONLINE     ONLINE          grac41        
ora.grac41.vip                 ONLINE     ONLINE          grac41        
ora.scan2.vip                  ONLINE     ONLINE          grac41

A closer look on Resource Dependencies

Database resource - Status:
#   crsctl status resource ora.grac4.db 
NAME=ora.grac4.db
TYPE=ora.database.type
TARGET=ONLINE          , ONLINE          , ONLINE
STATE=ONLINE on grac41, ONLINE on grac42, ONLINE on grac43
--> database is running on grac41

Database resource -  Configuration:
#  crsctl status resource ora.grac4.db -p  | egrep '^NAME|DEPEND'
NAME=ora.grac4.db
START_DEPENDENCIES=hard(ora.DATA.dg,ora.SSD.dg) 
                   weak(type:ora.listener.type,global:type:ora.scan_listener.type,uniform:ora.ons,global:ora.gns,ora.FRA.dg) 
                   pullup(ora.DATA.dg,ora.SSD.dg)
--> To start the DG DATA anD SSD datagroup must be already started  
    pullup entry means that ora.DATA.dg ora.SSD.dg are started automatically  whenever resource  ora.grac4.db  starts
    weak entry means that ora.grac4.db tries to start ora.listener.type,global:type:ora.scan_listener.type,uniform:ora.ons,global:ora.gns,ora.FRA.dg -
      if any of these resoruces fails to start it has not impact on starting resource  ora.grac4.db

Always use crsctl or srvctl to start/stop cluster resources

Using crsctl :
[root@grac41 bin]# crsctl stop resource  ora.LISTENER.lsnr  -n grac41
[root@grac41 bin]# crsctl start resource  ora.LISTENER.lsnr  -n grac41
[grid@grac41 ~]$  my_crs_stat | egrep 'LISTENER.ls|STATE_DETAILS|-----'
NAME                           TARGET     STATE           SERVER       STATE_DETAILS   
-------------------------      ---------- ----------      ------------ ------------------
ora.LISTENER.lsnr              ONLINE     ONLINE          grac41  

[root@grac41 bin]#  lsnrctl status listener
..
Service "grac4" has 1 instance(s).
  Instance "grac41", status READY, has 1 handler(s) for this service...
--> listener is working 

Using srvctl 
[grid@grac41 ~]$ srvctl stop listener  -n  grac41 -l LISTENER
[grid@grac41 ~]$ srvctl start listener  -n  grac41 -l LISTENER
[grid@grac41 ~]$  !my
my_crs_stat | egrep 'LISTENER.ls|STATE_DETAILS|-----'
NAME                           TARGET     STATE           SERVER       STATE_DETAILS   
-------------------------      ---------- ----------      ------------ ------------------
ora.LISTENER.lsnr              ONLINE     ONLINE          grac41 

Using lnsrctl 
[grid@grac41 ~]$ srvctl start listener  -n  grac41 -l LISTENER
[root@grac41 bin]#  lsnrctl status listener
..
Service "grac4" has 1 instance(s).
  Instance "grac41", status READY, has 1 handler(s) for this service...
[root@grac41 bin]# lsnrctl stop LISTENER
[grid@grac41 ~]$  my_crs_stat | egrep 'LISTENER.ls|STATE_DETAILS|-----'
NAME                           TARGET     STATE           SERVER       STATE_DETAILS   
-------------------------      ---------- ----------      ------------ ------------------
ora.LISTENER.lsnr              OFFLINE    OFFLINE         grac41  

[root@grac41 bin]#  lsnrctl start LISTENER
[grid@grac41 ~]$  my_crs_stat | egrep 'LISTENER.ls|STATE_DETAILS|-----'
NAME                           TARGET     STATE           SERVER       STATE_DETAILS   
-------------------------      ---------- ----------      ------------ ------------------
ora.LISTENER.lsnr              OFFLINE    OFFLINE         grac41  
--> Listener remain OFFLINE 

[root@grac41 bin]#  lsnrctl status LISTENER
--> No services where registered !

Oracle always recommend the use of the SRVCTL due to the fact that the SRVCTL does more, i.e. SRVCTL will 
perform a dependency analysis and will inform about cluster related problem if they exist, in a better way 
than the SQLPLUS which may just say that the instance is not starting. The SRVCTL command will try to start 
dependent resources (like the  vip/ons/listeners) in case they are not running.

The SRVCTL utility always do some kind of pre-start of the instances such as updating the
OCR information instead of waiting for the check script of the instance resource to detect this
instance startup and update the OCR.

Tracing status change of database, services and listener resoruces

$GRID_HOME/bin/oraagent.bin is the AGENT running the action script 

Tracefile location for databases, services:  crsd/oraagent_oracle/oraagent_oracle.log
Tracefile location for listners, ONS     :   crsd/oraagent_oracle/oraagent_oracle.log

Tracing an planned status change for datebase resource: ora.grac4.db 
[root@grac41 oraagent_oracle]# crsctl status  resource ora.grac4.db -f
NAME=ora.grac4.db
TYPE=ora.database.type
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
--> The agent binary is : $GRID_HOME/bin/oraagent.bin and the status is checked every second 

[grid@grac41 ~]$ crsctl stop resource ora.grac4.db -n grac41 -f
CRS-2673: Attempting to stop 'ora.grac4.grac41.svc' on 'grac41'
CRS-2677: Stop of 'ora.grac4.grac41.svc' on 'grac41' succeeded
CRS-2673: Attempting to stop 'ora.grac4.db' on 'grac41'
CRS-2677: Stop of 'ora.grac4.db' on 'grac41' succeeded

Review Tracefile:  crsd/oraagent_oracle/oraagent_oracle.log
[grid@grac41 ~]$ crsctl stop resource ora.grac4.db -n grac41 -f

2014-04-25 13:24:13.950: [    AGFW][872412928]{1:51811:6772} Agent received the message: AGENT_HB[Engine] ID 12293:36571
2014-04-25 13:24:14.290: [    AGFW][872412928]{1:51811:6817} Agent received the message: RESOURCE_STOP[ora.grac4.grac41.svc 1 1] ID 4099:36579
2014-04-25 13:24:14.290: [    AGFW][872412928]{1:51811:6817} Preparing STOP command for: ora.grac4.grac41.svc 1 1
2014-04-25 13:24:14.290: [    AGFW][872412928]{1:51811:6817} ora.grac4.grac41.svc 1 1 state changed from: ONLINE to: STOPPING
..
2014-04-25 13:24:14.446: [    AGFW][857704192]{1:51811:6817} Command: stop for resource: ora.grac4.grac41.svc 1 1 completed with status: SUCCESS
2014-04-25 13:24:14.448: [    AGFW][872412928]{1:51811:6817} Agent sending reply for: RESOURCE_STOP[ora.grac4.grac41.svc 1 1] ID 4099:36579
..
2014-04-25 13:24:14.452: [    AGFW][872412928]{1:51811:6817} ora.grac4.grac41.svc 1 1 state changed from: STOPPING to: OFFLINE
2014-04-25 13:24:14.452: [    AGFW][872412928]{1:51811:6817} Agent sending last reply for: RESOURCE_STOP[ora.grac4.grac41.svc 1 1] ID 4099:36579
2014-04-25 13:24:14.484: [    AGFW][872412928]{1:51811:6817} Agent received the message: RESOURCE_STOP[ora.grac4.db 1 1] ID 4099:36585
2014-04-25 13:24:14.484: [    AGFW][872412928]{1:51811:6817} Preparing STOP command for: ora.grac4.db 1 1
2014-04-25 13:24:14.484: [    AGFW][872412928]{1:51811:6817} ora.grac4.db 1 1 state changed from: ONLINE to: STOPPING
..
2014-04-25 13:24:40.531: [    AGFW][857704192]{1:51811:6817} Command: stop for resource: ora.grac4.db 1 1 completed with status: SUCCESS
2014-04-25 13:24:40.531: [CLSFRAME][962721568] TM [MultiThread] is changing desired thread # to 4. Current # is 3
2014-04-25 13:24:40.532: [    AGFW][872412928]{1:51811:6817} Agent sending reply for: RESOURCE_STOP[ora.grac4.db 1 1] ID 4099:36585
..
2014-04-25 13:24:40.570: [    AGFW][872412928]{1:51811:6817} ora.grac4.db 1 1 state changed from: STOPPING to: PLANNED_OFFLINE
2014-04-25 13:24:40.570: [    AGFW][872412928]{1:51811:6817} ora.grac4.db 1 1 state details has changed from: Open to: Instance Shutdown
2014-04-25 13:24:40.570: [    AGFW][872412928]{1:51811:6817} Agent sending last reply for: RESOURCE_STOP[ora.grac4.db 1 1] ID 4099:36585
2014-04-25 13:24:40.572: [    AGFW][853501696]{1:51811:6817} Agent has no resources to be monitored, Shutting down ..
2014-04-25 13:24:40.572: [    AGFW][853501696]{1:51811:6817} Agent sending message to PE: AGENT_SHUTDOWN_REQUEST[Proxy] ID 20486:2796
2014-04-25 13:24:40.577: [    AGFW][872412928]{1:51811:6817} Agent is shutting down.
2014-04-25 13:24:40.577: [    AGFW][872412928]{1:51811:6817} Agent is exiting with exit code: 1

Tracing an unplanned status change for resource  ora.LISTENER.lsnr

Check resource properties:
[grid@grac41 ~]$ crsctl status resource  ora.LISTENER.lsnr -f
NAME=ora.LISTENER.lsnr
TYPE=ora.listener.type
STATE=ONLINE
TARGET=ONLINE
ACTION_SCRIPT=%CRS_HOME%/bin/racgwrap%CRS_SCRIPT_SUFFIX%
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
ALIAS_NAME=ora.%CRS_CSS_NODENAME_LOWER_CASE%.LISTENER_%CRS_CSS_NODENAME_UPPER_CASE%.lsnr
AUTO_START=restore
CHECK_INTERVAL=60
--> The agent binary is : $GRID_HOME/bin/oraagent.bin and the status is checked every second 60 seconds

Every 60 seconds the listener status gets checked ( lsnrctl status LISTENER )
2014-04-25 14:15:22.372: [ora.LISTENER.lsnr][582964992]{1:51811:6685} [check] Utils:execCmd action = 3 flags = 38 ohome = (null) cmdname = lsnrctl. 
2014-04-25 14:15:22.477: [ora.LISTENER.lsnr][582964992]{1:51811:6685} [check] execCmd ret = 0
..
2014-04-25 14:16:22.377: [ora.LISTENER.lsnr][582964992]{1:51811:6685} [check] Utils:execCmd action = 3 flags = 38 ohome = (null) cmdname = lsnrctl. 
2014-04-25 14:16:22.481: [ora.LISTENER.lsnr][582964992]{1:51811:6685} [check] execCmd ret = 0
--> Return code is 0 

You can verify/test the status by running :
[grid@grac41 ~]$ lsnrctl status LISTENER 
...
  Instance "grac41", status READY, has 1 handler(s) for this service...
Service "grac4XDB" has 1 instance(s).
  Instance "grac41", status READY, has 1 handler(s) for this service...
The command completed successfully
[grid@grac41 ~]$ echo $status
0

Kill local listener
[root@grac41 oraagent_oracle]# ps -elf | grep tns
4 S daemon   13322     1  0  80   0 - 54027 poll_s 11:59 ?        00:00:00 /u01/app/11204/grid/bin/tnslsnr LISTENER -inherit
0 S grid     19482     1  0  80   0 - 54159 poll_s 08:14 ?        00:00:00 /u01/app/11204/grid/bin/tnslsnr LISTENER_SCAN2 -inherit
[root@grac41 oraagent_oracle]# kill -9 13322
-->  This triggers an unplanned resource status change for resource ora.LISTENER.lsnr on grac41

..
2014-04-25 14:30:37.956: [ora.LISTENER.lsnr][582964992]{1:51811:6939} [check] Utils:execCmd action = 3 flags = 38 ohome = (null) cmdname = lsnrctl. 
2014-04-25 14:30:39.683: [ora.LISTENER.lsnr][582964992]{1:51811:6939} [check] (:CLSN00010:)Utils:execCmd scls_process_join() uret 1

2014-04-25 14:30:39.683: [ora.LISTENER.lsnr][582964992]{1:51811:6939} [check] execCmd ret = 1
--> Listener status check  returns 1 instead of 0 - Agent frame work immediate restart the listener resource
2014-04-25 14:30:39.684: [    AGFW][1088186112]{1:51811:6939} ora.LISTENER.lsnr grac41 1 state changed from: ONLINE to: OFFLINE
2014-04-25 14:30:39.684: [    AGFW][1088186112]{0:17:25} Generating new Tint for unplanned state change. Original Tint: {1:51811:6939}
    --> Creating new TINT means we have a unplanned resoruce status change before 
2014-04-25 14:30:39.684: [    AGFW][1088186112]{0:17:25} Agent sending message to PE: RESOURCE_STATUS[Proxy] ID 20481:6019
2014-04-25 14:30:39.707: [    AGFW][1088186112]{0:17:25} Agent received the message: RESOURCE_START[ora.LISTENER.lsnr grac41 1] ID 4098:37563
2014-04-25 14:30:39.707: [    AGFW][1088186112]{0:17:25} Preparing START command for: ora.LISTENER.lsnr grac41 1
2014-04-25 14:30:39.707: [    AGFW][1088186112]{0:17:25} ora.LISTENER.lsnr grac41 1 state changed from: OFFLINE to: STARTING
--> Agent frame work changes listener status from: ONLINE to: OFFLINE and restarts listener 
    Agent frame work changes listener status from:  OFFLINE to: STARTING
..  
2014-04-25 14:30:43.178: [    AGFW][582964992]{0:17:25} Command: start for resource: ora.LISTENER.lsnr grac41 1 completed with status: SUCCESS
2014-04-25 14:30:43.178: [    AGFW][1088186112]{0:17:25} Agent sending reply for: RESOURCE_START[ora.LISTENER.lsnr grac41 1] ID 4098:37563
..
2014-04-25 14:30:43.306: [    AGFW][1088186112]{0:17:25} ora.LISTENER.lsnr grac41 1 state changed from: STARTING to: ONLINE
2014-04-25 14:30:43.306: [    AGFW][1088186112]{0:17:25} Started implicit monitor for [ora.LISTENER.lsnr grac41 1] interval=60000 delay=60000
2014-04-25 14:30:43.306: [    AGFW][1088186112]{0:17:25} Agent sending last reply for: RESOURCE_START[ora.LISTENER.lsnr grac41 1] ID 4098:37563

Delete a resource fails with CRS-2730

[grid@grac41 ~]$ crsctl delete  resource 'ora.OPENFILER_DG.dg'
CRS-2730: Resource 'ora.grac4.db' depends on resource 'ora.OPENFILER_DG.dg'

Check database dependencies
[oracle@grac41 trace]$ srvctl config database -d grac4 ( or use:  $ crsctl status resource ora.grac4.db -f )
Database unique name: grac4
Database name: grac4
Oracle home: /u01/app/oracle/product/11204/racdb
Oracle user: oracle
Spfile: +DATA/grac4/spfilegrac4.ora
Domain: 
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: grac4
Database instances: grac41,grac42,grac43
Disk Groups: DATA,SSD,OPENFILER_DG
Mount point paths: 
Services: grac41,grac42,grac43,report
Type: RAC
Database is administrator managed

Change diskgroup dependencies
[root@grac41 Desktop]# srvctl modify database -d grac4 -a 'DATA,SSD'
[root@grac41 Desktop]# srvctl config database -d grac4 
Database unique name: grac4
Database name: grac4
Oracle home: /u01/app/oracle/product/11204/racdb
Oracle user: oracle
Spfile: +DATA/grac4/spfilegrac4.ora
Domain: 
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: grac4
Database instances: grac41,grac42,grac43
Disk Groups: DATA,SSD
Mount point paths: 
Services: grac41,grac42,grac43,report
Type: RAC
Database is administrator managed

Now delete resource
[root@grac41 Desktop]#   crsctl delete  resource 'ora.OPENFILER_DG.dg' 

References:

  • 11.2.0.3 Grid Infrastructure diskmon Will be Offline by Default in Non-Exadata Environment (Doc ID 1346881.1)>

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>