Upgrade to GRID 11.2.0.4 from 11.2.0.3.11

Overiew

  • rootupgrade.sh (and root.sh for that matter) are restartable since 11.2.0.2 at least
  •  A great deal of logging is available in $GRID_HOME/cfgtoollogs/crsconfig/rootcrs_hostname.log    
    • Checkpoint file  : /u01/app/grid/Clusterware/ckptGridHA_grac31.xml
    • Log File         :  $GRID_HOME/cfgtoollogs/crsconfig/rootcrs_grac31.log

Note : This article simulates and fix a failed upgrade by having  account qosadmim deactivated –
      ( see Note 1577072.1 for details )

Verify that we are ready to UPGRADE with orachk

[oracle@grac31 ORACHECK225]$  ./orachk -u -o pre 
 Enter upgrade target version (valid versions are 11.2.0.3.0, 11.2.0.4.0, 12.1.0.1.0, 12.1.0.2.0):- 11.2.0.4.0 
CRS stack is running and CRS_HOME is not set. Do you want to set CRS_HOME to /u01/app/11203/grid?[y/n][y]
Checking ssh user equivalency settings on all nodes in cluster

Data collections completed. Checking best practices on grac31.
--------------------------------------------------------------------------------------
 INFO =>    Prior to Upgrade Verify NO Materialized Views Being Refreshed (Automatically or Manually) for on11203
 INFO =>    Special Considerations for SCAN and HAIP of 11.2.0.2 and above  Grid Infrastructure
 INFO =>    Minimizing Planned Downtime for Upgrades and Patching
 INFO =>    Upgrade Related References
 INFO =>    Lifetime Support Policy and Product Certification INFO
 INFO =>    Capture performance baseline, backup important configuration files and batch, cron, DBMS_JOBS and DBMS_SCHEDULER jobs
 INFO =>    Known Issues Integrating pre-11gR2 Databases with  Grid Infrastructure
 INFO =>    Be Aware of New Instant Client for Simplified Deployment
 INFO =>    Oracle E-Business Suite interoperability and migration resources
 INFO =>    Oracle Software Download Sites
 WARNING => One or More Column Names in ALL_TAB_COLUMNS table are Reserved Words for on11203
 WARNING => Some Users Needing Network ACLs for Oracle Utility Packages Found for on11203
 WARNING => Package unixODBC-2.2.14-11.el6-i686 is recommended but NOT installed
 WARNING => Package unixODBC-devel-2.2.14-11.el6-i686 is recommended but NOT installed
  INFO =>    Steps to execute rootupgrade.sh
 INFO =>    Save "Oracle Enterprise Manager Database Control" files and data with the emdwgrd Utility before upgrading  database.
 INFO =>    Information about ASM process parameter when its not set to default value
 INFO =>    Information about Grid Infrastructure software installation directory
 WARNING => Package unixODBC-devel-2.2.14-11.el6-x86_64 is recommended but NOT installed
---------------------------------------------------------------------------------
                      CLUSTERWIDE CHECKS
---------------------------------------------------------------------------------
Detailed report (html) - /home/oracle/ORACHECK225/orachk_grac31_on11203_092214_152522/orachk_grac31_on11203_092214_152522.html
UPLOAD(if required) - /home/oracle/ORACHECK225/orachk_grac31_on11203_092214_152522.zip

Check that qosadmin account is disabled which aborts/stops the upgrade process ( see  Doc ID 1577072.1 )
[root@grac31 Desktop]# qosctl qosadmin -listusers
AbstractLoginModule password:   <-- oracle112         
Sep 22, 2014 7:05:48 PM oracle.security.jazn.spi.xml.XMLRealmUser authenticate
INFO: User(jazn.com/qosadmin) is deactivated. AUTH FAILURE.
Sep 22, 2014 7:05:48 PM oracle.security.jazn.login.module.RealmLoginModule authenticate
SEVERE:         [RealmLoginModule] authentication failed

Note : The above error will abort the rootupgrade.sh script later on

Prepare and Install  new 11.2.0.4 GRID_HOME

[grid@grac31 ~]$  env | grep -i ORAC
ORACLE_BASE=/u01/app/grid
ORACLE_HOME=/u01/app/11204_upg/grid
[grid@grac31 ~]$ ls -ld /u01/app/oracle
drwxrwxr-x. 7 oracle oinstall 4096 Sep 19 17:37 /u01/app/oracle

Create the GRID_HOME directory 
[root@grac31 Desktop]# mkdir -p /u01/app/11204_upg/grid
[root@grac31 Desktop]# chown grid:oinstall /u01/app/11204_upg/grid
[root@grac31 Desktop]# chmod 775 /u01/app/11204_upg/grid
[root@grac31 Desktop]# ls -ld /u01/app/11204_upg/grid
drwxrwxr-x. 2 grid oinstall 4096 Sep 22 19:12 /u01/app/11204_upg/grid

Invoke Installer 

[grid@grac31 grid]$ ./runInstaller
--> Upgrade GRID Infrastructure or Oracle ASM

[root@grac31 Desktop]# /u01/app/11204_upg/grid/rootupgrade.sh
..
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11204_upg/grid/crs/install/crsconfig_params
Creating trace directory
Installing Trace File Analyzer

ASM upgrade has started on first node.
ASM upgrade has started on first node.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'grac31'
CRS-2673: Attempting to stop 'ora.crsd' on 'grac31'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'grac31'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'grac31'
..
CRS-2677: Stop of 'ora.drivers.acfs' on 'grac31' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'grac31' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'grac31'
CRS-2677: Stop of 'ora.gpnpd' on 'grac31' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'grac31' has completed
CRS-4133: Oracle High Availability Services has been stopped.
OLR initialization - successful
Replacing Clusterware entries in upstart
 clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.

Started to upgrade the Oracle Clusterware. This operation may take a few minutes.
Started to upgrade the CSS.
Started to upgrade the CRS.
The CRS was successfully upgraded.
Successfully upgraded the Oracle Clusterware.
Oracle Clusterware operating version was successfully set to 11.2.0.4.0

ASM upgrade has finished on last node.

Failed to perform J2EE (OC4J) Container Resource upgrade at /u01/app/11204_upg/grid/crs/install/crsconfig_lib.pm line 9323.
/u01/app/11204_upg/grid/perl/bin/perl -I/u01/app/11204_upg/grid/perl/lib -I/u01/app/11204_upg/grid/crs/install 
/u01/app/11204_upg/grid/crs/install/rootcrs.pl execution failed

Only ora.oc4j resource is OFFLINE
[root@grac31 Desktop]#  crs
NAME                           TARGET     STATE           SERVER       STATE_DETAILS   
-------------------------      ---------- ----------      ------------ ------------------
ora.oc4j                       OFFLINE    OFFLINE 

Fix the rootupgrade problem by following Note 1577072.1

For Case I - Upgrading Grid Infrastructure from 11.2.0.3 GI PSU 6 or higher to 11.2.0.4
Please perform the following workaround as grid user either before rootupgrade.sh is issued or after rootupgrade.sh fails with above error:

1. Activate the qosadmin user from 11.2.0.3 GRID_HOME as grid user:

grid@grac31 grid]$ env | grep GRID
GRID_HOME=/u01/app/11203/grid
[grid@grac31 grid]$ env | grep PATH
PATH=/u01/app/11203/grid/bin:.:/u01/app/11203/grid/bin:

Change qosadmin password to neworacle112
[grid@grac31 grid]$ qosctl qosadmin -setpasswd qosadmin oracle112 neworacle112
AbstractLoginModule password:       <<< enter   oracle112  
Please re-enter the password entered on the command line above
Enter Password:                    <<< enter   oracle112   
[grid@grac31 grid]$  qosctl qosadmin -listusers
AbstractLoginModule password:     <<< enter   neworacle112         
oc4jadmin
JtaAdmin
qosadmin

The next steps are not needed but I want to have back  ther original password in my test env
Change qosadmin password back to oracle112 ( here we use the default password again )
[grid@grac31 grid]$ qosctl qosadmin -setpasswd qosadmin  neworacle112 oracle112
AbstractLoginModule password:     <<< enter   neworacle112         
Please re-enter the password entered on the command line above
Enter Password:                  <<< enter   neworacle112         
[grid@grac31 grid]$   qosctl qosadmin -listusers
AbstractLoginModule password:     <<< enter   oracle112         
oc4jadmin
JtaAdmin
qosadmi

Rerun rootupgrade.sh from updated 11.2.0.4 CRS Home 
/u01/app/11204_upg/grid
[root@grac31 grid]# pwd
/u01/app/11204_upg/grid

[root@grac31 grid]# ./rootupgrade.sh
..
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11204_upg/grid/crs/install/crsconfig_params
Installing Trace File Analyzer
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-1115: Oracle Clusterware has already been upgraded.
ASM upgrade has finished on last node.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Related  rootcrs_grac31.log entries: Oracle Clusterware version on node [grac31] is [11.2.0.4.0]
>End Command output
2014-09-23 08:38:29: Version String passed is: Oracle Clusterware version on node [grac31] is [11.2.0.4.0]
2014-09-23 08:38:29: Version Info returned is : 11.2.0.4.0
2014-09-23 08:38:29: Got CRS softwareversion for grac31: 11.2.0.4.0
2014-09-23 08:38:29: The software version on grac31 is 11.2.0.4.0
2014-09-23 08:38:29: Invoking "/u01/app/11204_upg/grid/bin/srvctl upgrade model  -s 11.2.0.3.0 -d 11.2.0.4.0 -p first"
2014-09-23 08:38:29: trace file=/u01/app/11204_upg/grid/cfgtoollogs/crsconfig/srvmcfg0.log
2014-09-23 08:38:29: Executing /u01/app/11204_upg/grid/bin/srvctl upgrade model  -s 11.2.0.3.0 -d 11.2.0.4.0 -p first
2014-09-23 08:38:29: Executing cmd: /u01/app/11204_upg/grid/bin/srvctl upgrade model  -s 11.2.0.3.0 -d 11.2.0.4.0 -p first
2014-09-23 08:38:31: srvctl upgrade model -first  ... passed
2014-09-23 08:38:31: Running as user grid: /u01/app/11204_upg/grid/bin/srvctl stop oc4j
2014-09-23 08:38:31: s_run_as_user2: Running /bin/su grid -c ' /u01/app/11204_upg/grid/bin/srvctl stop oc4j '
2014-09-23 08:38:32: Removing file /tmp/filefdfk1C
2014-09-23 08:38:32: Successfully removed file: /tmp/filefdfk1C
2014-09-23 08:38:32: /bin/su exited with rc=2

2014-09-23 08:38:32: J2EE (OC4J) Container Resource Stop ... passed ...
2014-09-23 08:38:32: Running as user grid: /u01/app/11204_upg/grid/bin/srvctl disable oc4j
2014-09-23 08:38:32: s_run_as_user2: Running /bin/su grid -c ' /u01/app/11204_upg/grid/bin/srvctl disable oc4j '
2014-09-23 08:38:32: Removing file /tmp/file5FS4ZR
2014-09-23 08:38:32: Successfully removed file: /tmp/file5FS4ZR
2014-09-23 08:38:32: /bin/su exited with rc=1

2014-09-23 08:38:32: J2EE (OC4J) Container Resource Disable ... passed
2014-09-23 08:38:32: J2EE (OC4J) Container Resource stopped and disabled
2014-09-23 08:38:32: isLastNodeToUpgrade...
2014-09-23 08:38:32: isLastNodeToUpgrade: 1
2014-09-23 08:38:32: Executing /u01/app/11204_upg/grid/bin/crsctl set crs activeversion
2014-09-23 08:38:32: Executing cmd: /u01/app/11204_upg/grid/bin/crsctl set crs activeversion
2014-09-23 08:38:32: Command output:
>  CRS-1115: Oracle Clusterware has already been upgraded.
>End Command output
2014-09-23 08:38:32: /u01/app/11204_upg/grid/bin/crsctl set crs activeversion ... passed
2014-09-23 08:39:32: Rolling upgrade is set to 1
2014-09-23 08:39:32: End ASM rolling upgrade
2014-09-23 08:39:32: Executing as grid: /u01/app/11204_upg/grid/bin/asmca -silent -upgradeLocalASM -lastNode /u01/app/11203/grid
2014-09-23 08:39:32: Running as user grid: /u01/app/11204_upg/grid/bin/asmca -silent -upgradeLocalASM -lastNode /u01/app/11203/grid
2014-09-23 08:39:32:   Invoking "/u01/app/11204_upg/grid/bin/asmca -silent -upgradeLocalASM -lastNode /u01/app/11203/grid" as user "grid"
2014-09-23 08:39:32: Executing /bin/su grid -c "/u01/app/11204_upg/grid/bin/asmca -silent -upgradeLocalASM -lastNode /u01/app/11203/grid"
2014-09-23 08:39:32: Executing cmd: /bin/su grid -c "/u01/app/11204_upg/grid/bin/asmca -silent -upgradeLocalASM -lastNode /u01/app/11203/grid"
2014-09-23 08:39:45: Command output:
> 
>  ASM upgrade has finished on last node.
..

Rerun ./rootupgrade.sh a 3rd time  ( now we don’t see any CRS-1115 )

[root@grac31 grid]# ./rootupgrade.sh
..
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11204_upg/grid/crs/install/crsconfig_params
Installing Trace File Analyzer
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

 

Reference

  • GI Upgrade from 11.2.0.3.6+ to 11.2.0.4 or 12.1.0.1 Fails with User(qosadmin) is deactivated. AUTH FAILURE. (Doc ID 1577072.1)
  • How to Downgrade 11.2 Grid Infrastructure Standalone (Oracle Restart) After Successful or Failed Upgrade (Doc ID 1364412.1)

Leave a Reply

Your email address will not be published. Required fields are marked *