Install Patch Set 10.2.0.5 on top of 10.2.0.1 RAC base release ( CRS rolling upgrade )

Overview

  • Clusterware will have an active version and software version. The active version in my case is the 10.2.0,1  base release, and until the patch hasn’t completed (i.e. root.sh executed on last node) on all nodes that will remain the original. When root102.sh completes on the last node, the active version will match the software version
  • CRS is upgraded running a rolling upgrade

 

Check current CRS  and RDBMS status

[oracle@ract2 ~]$ crsctl query crs activeversion
CRS active version on the cluster is [10.2.0.1.0]
[oracle@ract2 ~]$ crsctl query crs softwareversion
CRS software version on node [ract2] is [10.2.0.1.0]

[oracle@ract2 ~]$  crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

Before applying the 10.2.0.5 patch set to the Clusterware home, run the following command on all the cluster nodes
# chmod -R +w $ORA_CRS_HOME/inventory/Templates/*

Check for invalid objects
SQL>  select object_name,status from dba_objects where status='INVALID';
no rows selected

Check for timezone version
SQL>  SELECT version FROM v$timezone_file;
   VERSION
----------
         2
As version != 4 you need to follow Actions for the DSTv4 update in the 10.2.0.5 patchset (Doc ID 1086400.1)

Verify RAC status with cluvfy ( always use newest cluvfy version and version switch like -r 10gR2 )

oracle@ract1 cluvfy12]$   ./bin/cluvfy  -version
12.1.0.1.0 Build 112713x8664
[oracle@ract1 cluvfy12]$   ./bin/cluvfy  stage -pre crsinst -r 10gR2   -n ract1,ract2,ract3

Prepare Clusterware for a Rolling Upgrade

Stop all Processes in node ract1
[oracle@ract1 ~]$ emctl stop dbconsole
[oracle@ract1 ~]$ srvctl stop instance -d RACT -i RACT1 -o immediate
[oracle@ract1 ~]$ lsnrctl stop  LISTENER_RACT1
[oracle@ract1 ~]$ srvctl stop asm -n ract1
[root@ract1 admin]#   srvctl stop nodeapps -n ract1

If database is running  on the same node relocate DB
[root@ract2 network-scripts]# crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.RACT.db    application    ONLINE    ONLINE    ract1
[root@ract2 network-scripts]#  crs_relocate ora.RACT.db
Attempting to stop `ora.RACT.db` on member `ract1`
Stop of `ora.RACT.db` on member `ract1` succeeded.
Attempting to start `ora.RACT.db` on member `ract2`
Start of `ora.RACT.db` on member `ract2` succeeded.

Backup your data

• Verify OCR and  Backup OCR
# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version                  :          2
Total space (kbytes)     :    1044104
Used space (kbytes)      :       5380
Available space (kbytes) :    1038724
ID                       : 1065530224
Device/File Name         : /dev/raw/raw1
Device/File integrity check succeeded
Device/File Name         : /dev/raw/raw2
Device/File integrity check succeeded
Cluster registry integrity check succeeded

Export OCR
# ocrconfig -export /restore/crs/ocr_backup.ocr -s online

• Voting Disk Backup
# crsctl query css votedisk
0.     0    /dev/raw/raw3
1.     0    /dev/raw/raw4
2.     0    /dev/raw/raw5
# dd if=/dev/raw/raw3 of=/restore/crs/votedisk1.dmp
# dd if=/dev/raw/raw4 of=/restore/crs/votedisk2.dmp
# dd if=/dev/raw/raw5 of=/restore/crs/votedisk3.dmp

Backup CLUSTER HOME, ORACLE HOME and OraInventory on all nodes
• Cluster Home and Oracle Homes [ALL Nodes]
# tar -czvf /restore/ORA_CRS_HOME.tar.gz /u01/app/oracle/product/crs
# tar -czvf /restore/ORACLE_HOME.tar.gz /u01/app/oracle/product/10.2/rac_db1

Locate OraInventory
# cat /etc/oraInst.loc
inventory_loc=/home/oracle/oraInventory
inst_group=oinstall
# tar -czvf /restore/ORA_INVENTORY.tar.gz /home/oracle/oraInventory
-> Note if your ASM is not installed with the RDBMS you need to backup your ASM ORACLE_HOME too

• Backup /etc/oracle
# tar -czvf /restore/ETC_ORACLE.tar.gz /etc/oracle
and save:
/etc/init.d/init.cssd
/etc/init.d/init.crs
/etc/init.d/init.crsd
/etc/init.d/init.evmd
/etc/inittab 

For more details check:
   10gR2/11gR1: What Files To Backup In Oracle Clusterware (CRS) Installation (Doc ID 754369.1)


--> In any case you should have a full OS backup before applying the patch

Verify CRS a last time
ot@ract1 ~]# crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....T1.inst application    OFFLINE   OFFLINE
ora....T2.inst application    ONLINE    ONLINE    ract2
ora....T3.inst application    ONLINE    ONLINE    ract3
ora.RACT.db    application    ONLINE    ONLINE    ract2
ora....SM1.asm application    OFFLINE   OFFLINE
ora....T1.lsnr application    OFFLINE   OFFLINE
ora.ract1.gsd  application    OFFLINE   OFFLINE
ora.ract1.ons  application    OFFLINE   OFFLINE
ora.ract1.vip  application    OFFLINE   OFFLINE
ora....SM2.asm application    ONLINE    ONLINE    ract2
ora....T2.lsnr application    ONLINE    ONLINE    ract2
ora.ract2.gsd  application    ONLINE    ONLINE    ract2
ora.ract2.ons  application    ONLINE    ONLINE    ract2
ora.ract2.vip  application    ONLINE    ONLINE    ract2
ora....SM3.asm application    ONLINE    ONLINE    ract3
ora....T3.lsnr application    ONLINE    ONLINE    ract3
ora.ract3.gsd  application    ONLINE    ONLINE    ract3
ora.ract3.ons  application    ONLINE    ONLINE    ract3
ora.ract3.vip  application    ONLINE    ONLINE    ract3

--> Now its time to backup your database using RMAN

Install 10.2.0.5 clusterware software

[oracle@ract1 Disk1]$  ./runInstaller
--> Select your  CRS Oracle_HOME
OUI prompts:
The installer has detected that your Cluster Ready Services (CRS) installation is distributed across the following nodes:
ract1
ract2
ract3
Because the software consists of local identical copies distributed across each of the nodes in the cluster, it is possible to patch your CRS installation in a rolling manner, one node at a time.
To complete the installation of this patchset, you must perform the following tasks on each node:

1.    Log in as the root user.
2.    As the root user, perform the following tasks:
a.    Shutdown the CRS daemons by issuing the following command:
/u01/app/oracle/product/crs/bin/crsctl stop crs
b.    Run the shell script located at:
/u01/app/oracle/product/crs/install/root102.sh
This script will automatically start the CRS daemons on the
patched node upon completion.
3.    After completing this procedure, proceed to the next node and repeat.

Start with node ract1
[root@ract1 ~]#  /u01/app/oracle/product/crs/bin/crsctl stop crs
Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
[root@ract1 ~]# /u01/app/oracle/product/crs/install/root102.sh
Creating pre-patch directory for saving pre-patch clusterware files
Completed patching clusterware files to /u01/app/oracle/product/crs
Relinking some shared libraries.
Relinking of patched files is complete.
WARNING: directory '/u01/app/oracle/product' is not owned by root
WARNING: directory '/u01/app/oracle' is not owned by root
WARNING: directory '/u01/app' is not owned by root
WARNING: directory '/u01' is not owned by root
Preparing to recopy patched init and RC scripts.
Recopying init and RC scripts.
Startup will be queued to init within 30 seconds.
Starting up the CRS daemons.
Waiting for the patched CRS daemons to start.
This may take a while on some systems.
.
10205 patch successfully applied.
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully deleted 1 values from OCR.
Successfully deleted 1 keys from OCR.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: ract1 ract1int ract1
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
clscfg -upgrade completed successfully
Creating '/u01/app/oracle/product/crs/install/paramfile.crs' with data used for CRS configuration
Setting CRS configuration values in /u01/app/oracle/product/crs/install/paramfile.crs

Verify CRS version on ract1
[root@ract1 ~]# crsctl query crs activeversion
CRS active version on the cluster is [10.2.0.1.0]
[root@ract1 ~]#   crsctl query crs softwareversion
CRS software version on node [ract1] is [10.2.0.5.0]

Run root102.sh on ract2
[root@ract2 ~]#  /u01/app/oracle/product/crs/bin/crsctl stop crs
[root@ract2 ~]#  /u01/app/oracle/product/crs/install/root102.sh

Check CRS status after running root102.sh on ract2
[root@ract1 ~]#  crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....T1.inst application    ONLINE    ONLINE    ract1
ora....T2.inst application    ONLINE    ONLINE    ract2
ora....T3.inst application    ONLINE    ONLINE    ract3
ora.RACT.db    application    ONLINE    ONLINE    ract3
ora....SM1.asm application    ONLINE    ONLINE    ract1
ora....T1.lsnr application    ONLINE    ONLINE    ract1
ora.ract1.gsd  application    ONLINE    ONLINE    ract1
ora.ract1.ons  application    ONLINE    ONLINE    ract1
ora.ract1.vip  application    ONLINE    ONLINE    ract1
ora....SM2.asm application    ONLINE    ONLINE    ract2
ora....T2.lsnr application    ONLINE    ONLINE    ract2
ora.ract2.gsd  application    ONLINE    ONLINE    ract2
ora.ract2.ons  application    ONLINE    ONLINE    ract2
ora.ract2.vip  application    ONLINE    ONLINE    ract2
ora....SM3.asm application    ONLINE    ONLINE    ract3
ora....T3.lsnr application    ONLINE    ONLINE    ract3
ora.ract3.gsd  application    ONLINE    ONLINE    ract3
ora.ract3.ons  application    ONLINE    ONLINE    ract3
ora.ract3.vip  application    ONLINE    ONLINE    ract3
--> Again after running root102.sh  all nodes are active ....

Now running root102.sh on the last node
[root@ract3 ~]#   /u01/app/oracle/product/crs/bin/crsctl stop crs
[root@ract3 ~]# /u01/app/oracle/product/crs/install/root102.sh

[root@ract3 ~]#  crsctl query crs activeversion
CRS active version on the cluster is [10.2.0.5.0]
[root@ract3 ~]#  crsctl query crs softwareversion
CRS software version on node [ract3] is [10.2.0.5.0]

--> CRS activersion is now 10.2.0.5.0 as root102.sh was run sucessfully on our last node!

Install 10.2.0.5 RDBMS software

Stopping All Processes for an Oracle RAC Installation
# srvctl stop database -d RACT
# srvctl stop asm -n  ract1
# srvctl stop asm -n  ract2
# srvctl stop asm -n  ract3
# srvctl stop listener -n ract1
# srvctl stop listener -n ract2
# srvctl stop listener -n ract3

Check CRS status
[root@ract1 ~]#  crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....T1.inst application    OFFLINE   OFFLINE
ora....T2.inst application    OFFLINE   OFFLINE
ora....T3.inst application    ONLINE    OFFLINE
ora.RACT.db    application    OFFLINE   OFFLINE
ora....SM1.asm application    OFFLINE   OFFLINE
ora....T1.lsnr application    OFFLINE   OFFLINE
ora.ract1.gsd  application    ONLINE    ONLINE    ract1
ora.ract1.ons  application    ONLINE    ONLINE    ract1
ora.ract1.vip  application    ONLINE    ONLINE    ract1
ora....SM2.asm application    OFFLINE   OFFLINE
ora....T2.lsnr application    OFFLINE   OFFLINE
ora.ract2.gsd  application    ONLINE    ONLINE    ract2
ora.ract2.ons  application    ONLINE    ONLINE    ract2
ora.ract2.vip  application    ONLINE    ONLINE    ract2
ora....SM3.asm application    OFFLINE   OFFLINE
ora....T3.lsnr application    OFFLINE   OFFLINE
ora.ract3.gsd  application    ONLINE    ONLINE    ract3
ora.ract3.ons  application    ONLINE    ONLINE    ract3
ora.ract3.vip  application    ONLINE    ONLINE    ract3

Run OUI
[oracle@ract1 Disk1]$  ./runInstaller
Select : OraDb10g_home1  -> /u01/app/oracle/product/10.2/rac_db1

Upgrade Database  to 10.2.0.5

[oracle@ract1 ~]$ sqlplus / as sysdba
SQL> STARTUP NOMOUNT
SQL> ALTER SYSTEM SET CLUSTER_DATABASE=FALSE SCOPE=spfile;
SQL> SHUTDOWN
SQL> STARTUP UPGRADE
SQL> SPOOL patch.log
SQL> @?/rdbms/admin/catupgrd.sql
SQL> SPOOL OFF

Check patch.log for errors:
[root@ract1 Desktop]# grep ORA- -/patch.log
DOC>    The following statement will cause an "ORA-01722: invalid number"
DOC>    The following statement will cause an "ORA-01722: invalid number"
DOC>   The following statement will cause an "ORA-01722: invalid number"
DOC>    The following statements will cause an "ORA-01722: invalid number"
DOC>    The following statement will cause an "ORA-01722: invalid number"
DOC>    The following PL/SQL block will cause an ORA-20000 error and
DOC>    The following PL/SQL block will cause an ORA-20000 error and
ORA-29558: JAccelerator (NCOMP) not installed. Refer to Install Guide for
ORA-06512: at "SYS.DBMS_JAVA", line 236
SQL> !grep ORA- patch.log
ORA-00923: FROM keyword not found where expected

If you get a similiar output like above output you are save.
If not  rerun the catupgrd.sql script after correcting any problems.

Restart the database:
SQL> SHUTDOWN IMMEDIATE
SQL> STARTUP
Run the utlrp.sql script to recompile all invalid PL/SQL packages
SQL> @?/rdbms/admin/utlrp.sql

Check JAVA/PLSQL package status :
SQL>  select object_name,status from dba_objects where status='INVALID';
no rows selected

SQL>  select COMP_NAME,VERSION,STATUS  from dba_registry;
COMP_NAME                           VERSION                        STATUS
----------------------------------- ------------------------------ -----------
Oracle Enterprise Manager           10.2.0.5.0                     VALID
Spatial                             10.2.0.5.0                     VALID
Oracle interMedia                   10.2.0.5.0                     VALID
OLAP Catalog                        10.2.0.5.0                     VALID
Oracle XML Database                 10.2.0.5.0                     VALID
Oracle Text                         10.2.0.5.0                     VALID
Oracle Expression Filter            10.2.0.5.0                     VALID
Oracle Rule Manager                 10.2.0.5.0                     VALID
Oracle Workspace Manager            10.2.0.5.0                     VALID
Oracle Data Mining                  10.2.0.5.0                     VALID
Oracle Database Catalog Views       10.2.0.5.0                     VALID
Oracle Database Packages and Types  10.2.0.5.0                     VALID
JServer JAVA Virtual Machine        10.2.0.5.0                     VALID
Oracle XDK                          10.2.0.5.0                     VALID
Oracle Database Java Packages       10.2.0.5.0                     VALID
OLAP Analytic Workspace             10.2.0.5.0                     VALID
Oracle OLAP API                     10.2.0.5.0                     VALID
Oracle Real Application Clusters    10.2.0.5.0                     VALID

Set the CLUSTER_DATABASE initialization parameter to TRUE:
SQL> ALTER SYSTEM SET CLUSTER_DATABASE=TRUE SCOPE=spfile;
Restart the database:
SQL> SHUTDOWN IMMEDIATE
SQL> STARTUP

Start listener , ASM and RDBMS instances as needed
[oracle@ract1 ~]$ srvctl start listener -n ract2
[oracle@ract1 ~]$ srvctl start asm -n ract2
[oracle@ract1 ~]$ srvctl  start instance -d RACT -i RACT2
[oracle@ract1 ~]$ srvctl start listener -n ract3
[oracle@ract1 ~]$ srvctl start asm -n ract3
[oracle@ract1 ~]$ srvctl  start instance -d RACT -i RACT3

Check CRS status after RDBMS upgrade
[oracle@ract1 ~]$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....T1.inst application    ONLINE    ONLINE    ract1
ora....T2.inst application    ONLINE    ONLINE    ract2
ora....T3.inst application    ONLINE    ONLINE    ract3
ora.RACT.db    application    ONLINE    ONLINE    ract2
ora....SM1.asm application    ONLINE    ONLINE    ract1
ora....T1.lsnr application    ONLINE    ONLINE    ract1
ora.ract1.gsd  application    ONLINE    ONLINE    ract1
ora.ract1.ons  application    ONLINE    ONLINE    ract1
ora.ract1.vip  application    ONLINE    ONLINE    ract1
ora....SM2.asm application    ONLINE    ONLINE    ract2
ora....T2.lsnr application    ONLINE    ONLINE    ract2
ora.ract2.gsd  application    ONLINE    ONLINE    ract2
ora.ract2.ons  application    ONLINE    ONLINE    ract2
ora.ract2.vip  application    ONLINE    ONLINE    ract2
ora....SM3.asm application    ONLINE    ONLINE    ract3
ora....T3.lsnr application    ONLINE    ONLINE    ract3
ora.ract3.gsd  application    ONLINE    ONLINE    ract3
ora.ract3.ons  application    ONLINE    ONLINE    ract3
ora.ract3.vip  application    ONLINE    ONLINE    ract3

SQL> select inst_id, instance_name, host_name,  status, to_char(startup_time, 'DD-MON-YYYY HH24:MI:SS') startup_time,
version from gv$instance order by inst_id;
INST_ID INSTANCE_NAME    HOST_NAME            STATUS       STARTUP_TIME         VERSION
---------- ---------------- -------------------- ------------ -------------------- --------------------
1 RACT1            ract1.example.com    OPEN         15-APR-2014 19:35:35 10.2.0.5.0
2 RACT2            ract2.example.com    OPEN         15-APR-2014 19:36:36 10.2.0.5.0
5 RACT3            ract3.example.com    OPEN         15-APR-2014 19:38:12 10.2.0.5.0

Reference

  • 10gR2/11gR1: What Files To Backup In Oracle Clusterware (CRS) Installation (Doc ID 754369.1)

One thought on “Install Patch Set 10.2.0.5 on top of 10.2.0.1 RAC base release ( CRS rolling upgrade )”

  1. Dear Helmut,

    Very nicely explained article.

    I have one doubt though. in the section “Install 10.2.0.5 clusterware software”, have you performed rolling upgrade ?

    I am not sure if it can be called rolling upgrade as you are installing patchset in all the nodes in one go.

    Please clarify.

    Thanks.

Leave a Reply

Your email address will not be published. Required fields are marked *