How to fix a RAC database startup problem with orachk in 5 minutes ?

New TFA Collector bundles ORAchk and other RAC tools

TFA Collector Features

Installation of TFA Lite /  ORACHK

Note orachk is now bundled with TFA collector  
Use the MOS article below to download TFA collector 
    TFA Collector - Tool for Enhanced Diagnostic Gathering (Doc ID 1513912.2)

Extract and install orachk as root user 
root@grac41 t]# ls
installTFALite  TFACollectorDocV121250.pdf  TFALite_121250.zip
[root@grac41 t]# ./installTFALite
Starting TFA installation
Enter a location for installing TFA (/tfa will be appended if not supplied) [/home/oracle/t/tfa]: 
/home/oracle/TFA
...
---------------------------------.
|            TFA Users            |
+-----------+-----------+---------+
| User Name | User Type | Status  |
+-----------+-----------+---------+
| asmadmin  | GROUP     | Allowed |
| grid      | USER      | Allowed |
| oinstall  | GROUP     | Allowed |
| oracle    | USER      | Allowed |
'-----------+-----------+---------'
Summary of TFA Installation:
.------------------------------------------------------------.
|                           grac41                           |
+---------------------+--------------------------------------+
| Parameter           | Value                                |
+---------------------+--------------------------------------+
| Install location    | /home/oracle/TFA/tfa/grac41/tfa_home |
| Repository location | /home/oracle/TFA/tfa/repository      |
| Repository usage    | 0 MB out of 2982 MB                  |
'---------------------+--------------------------------------'
.------------------------------------------------------------.
|                           grac42                           |
+---------------------+--------------------------------------+
| Parameter           | Value                                |
+---------------------+--------------------------------------+
| Install location    | /home/oracle/TFA/tfa/grac42/tfa_home |
| Repository location | /home/oracle/TFA/tfa/repository      |
| Repository usage    | 0 MB out of 2982 MB                  |
'---------------------+--------------------------------------'
.------------------------------------------------------------.
|                           grac43                           |
+---------------------+--------------------------------------+
| Parameter           | Value                                |
+---------------------+--------------------------------------+
| Install location    | /home/oracle/TFA/tfa/grac43/tfa_home |
| Repository location | /home/oracle/TFA/tfa/repository      |
| Repository usage    | 0 MB out of 2982 MB                  |
'---------------------+--------------------------------------'
TFA is successfully installed...

 

Problem description

 
Either starting RAC instance with sqlplus or srvctl fails with errors :  

[oracle@grac43 dbs]$  sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Tue Jun 23 13:01:15 2015
Copyright (c) 1982, 2013, Oracle.  All rights reserved.
Connected to an idle instance.
SQL>  startup nomount
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/grac4/spfilegrac4.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/grac4/spfilegrac4.ora
ORA-01034: ORACLE not available
ORA-27123: unable to attach to shared memory segment
Linux-x86_64 Error: 13: Permission denied
Additional information: 26
Additional information: 229382

Run Orack and review the HTML file for Errors and Warnings


[root@grac41 orachk]# ./orachk -v
ORACHK  VERSION: 12.1.0.2.3_20150305

[root@grac41 orachk]# ./orachk
CRS stack is running and CRS_HOME is not set. Do you want to set CRS_HOME to /u01/app/11204/grid?[y/n][y]


Check generated HTML file for WARNINGS and ERRORS 
..
Cluster Wide Status 
WARNING => RDBMS software owner UID does not match across cluster

DATA FROM GRAC41 - RDBMS SOFTWARE OWNER UID ACROSS CLUSTER 
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),500(vboxsf),54322(dba),506(asmdba)

DATA FROM GRAC42 - RDBMS SOFTWARE OWNER UID ACROSS CLUSTER 
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),500(vboxsf),54322(dba),506(asmdba)

DATA FROM GRAC43 - RDBMS SOFTWARE OWNER UID ACROSS CLUSTER 
uid=501(grid) gid=54321(oinstall) groups=54321(oinstall),500(vboxsf),54322(dba),504(asmadmin),506(asmdba),507(asmoper) 

-> Here we see easily can see  that uid on GRAC43 is wrong  

Fix the error and verify database status  
[root@grac43 bin]# chown oracle oracle
[root@grac43 bin]# chmod 6751 oracle
[root@grac43 bin]# ls -l oracle
-rwsr-s--x. 1 oracle asmadmin 239732402 Jul 21  2014 oracle
[grid@grac43 ~]$ srvctl start instance -d grac4 -i grac43
[oracle@grac43 dbs]$ srvctl status database -d grac4
Instance grac41 is running on node grac41
Instance grac42 is running on node grac42
Instance grac43 is running on node grac43



Leave a Reply

Your email address will not be published. Required fields are marked *