Debug problems after a failed patch installation with opatch auto

Opatch  Log Files

  •  Main log file:    /u01/app/11204/grid/cfgtoollogs/opatchauto2014-07-21_10-23-50.log
  •  Configuration log file with details about CRS commands:  /u01/app/11204/grid/cfgtoollogs/opatchauto2014-07-21_10-23-50.report.log

Running Opatch auto

[root@grac41 Desktop]# $GRID_HOME/OPatch/opatch auto /PATCHES/Patch112043/18706472 -ocmrf  $GRID_HOME/OPatch/ocm.rsp
Executing /u01/app/11204/grid/perl/bin/perl /u01/app/11204/grid/OPatch/crs/patch11203.pl -patchdir /PATCHES/Patch112043 
           -patchn 18706472 -ocmrf /u01/app/11204/grid/OPatch/ocm.rsp
           -paramfile /u01/app/11204/grid/crs/install/crsconfig_params
This is the main log file: /u01/app/11204/grid/cfgtoollogs/opatchauto2014-07-21_10-23-50.log
This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/11204/grid/cfgtoollogs/opatchauto2014-07-21_10-23-50.report.log

2014-07-21 10:23:50: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/11204/grid/crs/install/crsconfig_params
Stopping RAC /u01/app/oracle/product/11204/racdb ...
Stopped RAC /u01/app/oracle/product/11204/racdb successfully
patch /PATCHES/Patch112043/18706472/18522509  apply successful for home  /u01/app/oracle/product/11204/racdb 
patch /PATCHES/Patch112043/18706472/18522515/custom/server/18522515  apply successful for home  /u01/app/oracle/product/11204/racdb 

Stopping CRS...
The Oracle Grid Infrastructure stack failed to stop.
Resetting the status, starting CRS...
ERROR: Failed to stop CRS.Refer log file for more details.
--> Patching CRS was unsucessfull 

Check Main Install Log file
2014-07-21 10:27:20: postpatch execution for DB home ... success
2014-07-21 10:27:20: Processing oracle home /u01/app/11204/grid
2014-07-21 10:27:20: Opening file /etc/oracle/ocr.loc
2014-07-21 10:27:20: Value (false) is set for key=local_only
2014-07-21 10:27:20: Home type of /u01/app/11204/grid is CRS
2014-07-21 10:27:20: Unlock crshome...
...
2014-07-21 10:33:23: /u01/app/11204/grid/bin/crsctl stop crs -f
2014-07-21 10:33:23: Executing cmd: /u01/app/11204/grid/bin/crsctl check cluster -n grac41
...
CRS-2799: Failed to shut down resource 'procwatcher' on 'grac41'
CRS-2794: Shutdown of Cluster Ready Services-managed resources on 'grac41' has failed
CRS-5022: Stop of resource "ora.crsd" failed: current state is "UNKNOWN"
CRS-2675: Stop of 'ora.crsd' on 'grac41' failed
CRS-2799: Failed to shut down resource 'ora.crsd' on 'grac41'
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'grac41' has failed
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.
################################################################
#You must kill processes or reboot the system to properly      #
#cleanup the processes started by Oracle Grid Infrastructure   #
################################################################
The Oracle Grid Infrastructure stack failed to stop.
You should stop the stack with 'crsctl stop crs' and rerun the command

..
2014-07-21 10:35:25: ERROR: Failed to stop CRS.Refer log file for more details.

Configuration Log:
***********  Configuration Data  ***********
* It shows only those targets that will be patched in this session *
crs_home=/u01/app/11204/grid    version=11.2.0.4.0    owner=grid      opatch_ver=11.2.0.3.6
rac_home=/u01/app/oracle/product/11204/racdb    version=11.2.0.4.0    owner=oracle      opatch_ver=11.2.0.3.6
*********** Steps to be executed as owner unless specified as root ***********
...
19: /PATCHES/Patch112043/18706472/18522515/custom/server/18522515/custom/scripts/postpatch.sh 
      -dbhome /u01/app/oracle/product/11204/racdb
20: /u01/app/11204/grid/crs/install/rootcrs.pl -unlock : run as root

--> CRS patching failed running last step in CRS Configuratuon file :  rootcrs.pl -unlock  
     rootcrs.pl -unlock failed running :   /u01/app/11204/grid/bin/crsctl stop crs -f
       /u01/app/11204/grid/bin/crsctl stop crs -f fails to stop the procwatcher resource 

Try to stop clusterware manually :
[root@grac41 Desktop]#  /u01/app/11204/grid/bin/crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'grac41'
CRS-2673: Attempting to stop 'ora.crsd' on 'grac41'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'grac41'
CRS-2679: Attempting to clean 'procwatcher' on 'grac41'
CRS-2679: Attempting to clean 'procwatcher' on 'grac41'
CRS-2680: Clean of 'procwatcher' on 'grac41' failed
CRS-5809: Failed to execute 'ACTION_SCRIPT' value of '/u01/app/11204/grid/log/procwatcher/prw.sh' for 'procwatcher'. Error information 
   'cmd /u01/app/11204/grid/log/procwatcher/prw.sh not found', Category : -2, OS error : 2
CRS-2680: Clean of 'procwatcher' on 'grac41' failed
CRS-2799: Failed to shut down resource 'procwatcher' on 'grac41'
CRS-2794: Shutdown of Cluster Ready Services-managed resources on 'grac41' has failed
CRS-5022: Stop of resource "ora.crsd" failed: current state is "UNKNOWN"
CRS-2675: Stop of 'ora.crsd' on 'grac41' failed
CRS-2799: Failed to shut down resource 'ora.crsd' on 'grac41'
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'grac41' has failed
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.
--- > Following errors need to be fixed  
   CRS-2799: Failed to shut down resource 'procwatcher' on 'grac41'
   CRS-2794: Shutdown of Cluster Ready Services-managed resources on 'grac41' has failed

Fix the problem with missing procwatcher resource 
As user grid run 
$  scp grac42:/u01/app/11204/grid/log/procwatcher/prw.sh /u01/app/11204/grid/log/procwatcher
$  chmod  755 /u01/app/11204/grid/log/procwatcher/prw.sh 

Now test  CRS stutdown 
[root@grac41 Desktop]# /u01/app/11204/grid/bin/crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'grac41'
..
CRS-4133: Oracle High Availability Services has been stopped.

As user root run the  failing step  again. 
[root@grac41 Desktop]# /u01/app/11204/grid/crs/install/rootcrs.pl -unlock  
Using configuration parameter file: /u01/app/11204/grid/crs/install/crsconfig_params
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
Successfully unlock /u01/app/11204/grid

After fixing the CRS shutdown problem rerun opatch auto again 
-->  it should not install any remaining not ye installed patches 
$GRID_HOME/OPatch/opatch auto /PATCHES/Patch112043/18706472 -ocmrf  $GRID_HOME/OPatch/ocm.rsp

Verify patch installation with 
[oracle@grac41 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory
grid@grac41 ~]$   $GRID_HOME/OPatch/opatch lsinventory

Leave a Reply

Your email address will not be published. Required fields are marked *