Backup current OLR and all available OLR backups – don’t miss this step
# ocrconfig -local -manualbackup grac41 2014/05/20 12:22:20 /u01/app/11204/grid/cdata/grac41/backup_20140520_122220.olr Verify available backups # ocrconfig -local -showbackup grac41 2014/05/20 12:22:20 /u01/app/11204/grid/cdata/grac41/backup_20140520_122220.olr .. # ls -l /u01/app/11204/grid/cdata/grac41/* -rw-------. 1 root root 6803456 Sep 12 2013 /u01/app/11204/grid/cdata/grac41/backup_20130912_130457.olr -rw-------. 1 root root 7016448 May 20 09:08 /u01/app/11204/grid/cdata/grac41/backup_20140520_090842.olr -rw-------. 1 root root 6807552 May 20 12:22 /u01/app/11204/grid/cdata/grac41/backup_20140520_122220.olr
Restore a OLR from backup
Shutdown CRS and check that ohasd.bin process is not running and restore OLR # crsctl stop crs # ps -elf | egrep "PID|d.bin|ohas|oraagent|orarootagent|cssdagent|cssdmonitor" | grep -v grep F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 4 S root 4494 1 0 80 0 - 2846 wait 10:04 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run # ocrconfig -local -restore /u01/app/11204/grid/cdata/grac41/backup_20140520_090842.olr # crsctl stop crs
Recover OLR after PROCL-26. PROCL-5 error and no OLR backups available
Tracefile: # crsctl start crs TraceFileName: ./client/ocrcheck_27844.log 2014-05-20 09:38:48.918: [OCRCHECK][2768283392]Failed to access OCR repository. Retval [26] Error [PROCL-26: Error while accessing the physical storage] TraceFileName: ./gpnpd/gpnpd_1.log 2014-05-19 13:23:19.100: [ GPNP][85341984]procr_open_key_ext: OLR error message : PROCL-5: User does not have permission to perform a local registry operation on this key. Authentication error [User does not have permission to perform this operation] [0] Details of PROCL-5 from ./gpnpd/gpnpd_1.log .. [ CLWAL][3606726432]clsw_Initialize: OLR initlevel [70000] [ OCRAPI][3606726432]a_init:10: AUTH LOC [/u01/app/11204/grid/srvm/auth] [ OCRMSG][3606726432]prom_init: Successfully registered comp [OCRMSG] in clsd. 2014-05-20 07:57:51.118: [ OCRAPI][3606726432]a_init:11: Messaging init successful. [ OCRCLI][3606726432]oac_init: Successfully registered comp [OCRCLI] in clsd. 2014-05-20 07:57:51.118: [ OCRCLI][3606726432]proac_con_init: Local listener using IPC. [(ADDRESS=(PROTOCOL=ipc)(KEY=procr_local_conn_0_PROL))] 2014-05-20 07:57:51.119: [ OCRCLI][3606726432]proac_con_init: Successfully connected to the server 2014-05-20 07:57:51.119: [ OCRCLI][3606726432]proac_con_init: Post sema. Con count [1] 2014-05-20 07:57:51.120: [ OCRAPI][3606726432]a_init:12: Client init successful. 2014-05-20 07:57:51.120: [ OCRAPI][3606726432]a_init:21: OCR init successful. Init Level [7] 2014-05-20 07:57:51.120: [ OCRAPI][3606726432]a_init:2: Init Level [7] 2014-05-20 07:57:51.132: [ OCRCLI][3606726432]proac_con_init: Post sema. Con count [2] [ clsdmt][3595089664]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=grac41DBG_GPNPD)) 2014-05-20 07:57:51.133: [ clsdmt][3595089664]PID for the Process [31034], connkey 10 2014-05-20 07:57:51.133: [ clsdmt][3595089664]Creating PID [31034] file for home /u01/app/11204/grid host grac41 bin gpnp to /u01/app/11204/grid/gpnp/init/ 2014-05-20 07:57:51.133: [ clsdmt][3595089664]Writing PID [31034] to the file [/u01/app/11204/grid/gpnp/init/grac41.pid] 2014-05-20 07:57:52.108: [ GPNP][3606726432]clsgpnpd_validateProfile: [at clsgpnpd.c:2919] GPnPD taken cluster name 'grac4' 2014-05-20 07:57:52.108: [ GPNP][3606726432]clsgpnpd_openLocalProfile: [at clsgpnpd.c:3477] Got local profile from file cache provider (LCP-FS). 2014-05-20 07:57:52.111: [ GPNP][3606726432]clsgpnpd_openLocalProfile: [at clsgpnpd.c:3532] Got local profile from OLR cache provider (LCP-OLR). 2014-05-20 07:57:52.113: [ GPNP][3606726432]procr_open_key_ext: OLR api procr_open_key_ext failed for key SYSTEM.GPnP.profiles.peer.pending 2014-05-20 07:57:52.113: [ GPNP][3606726432]procr_open_key_ext: OLR current boot level : 7 2014-05-20 07:57:52.113: [ GPNP][3606726432]procr_open_key_ext: OLR error code : 5 2014-05-20 07:57:52.126: [ GPNP][3606726432]procr_open_key_ext: OLR error message : PROCL-5: User does not have permission to perform a local registry operation on this key. Authentication error [User does not have permission to perform this operation] [0] 2014-05-20 07:57:52.126: [ GPNP][3606726432]clsgpnpco_ocr2profile: [at clsgpnpco.c:578] Result: (58) CLSGPNP_OCR_ERR. Failed to open requested OLR Profile. 2014-05-20 07:57:52.127: [ GPNP][3606726432]clsgpnpd_lOpen: [at clsgpnpd.c:1734] Listening on ipc://GPNPD_grac41 2014-05-20 07:57:52.127: [ GPNP][3606726432]clsgpnpd_lOpen: [at clsgpnpd.c:1743] GIPC gipcretFail (1) gipcListen listen failure on 2014-05-20 07:57:52.127: [ default][3606726432]GPNPD failed to start listening for GPnP peers. 2014-05-20 07:57:52.135: [ GPNP][3606726432]clsgpnpd_term: [at clsgpnpd.c:1344] STOP GPnPD terminating. Closing connections... 2014-05-20 07:57:52.137: [ default][3606726432]clsgpnpd_term STOP terminating. 2014-05-20 07:57:53.136: [ OCRAPI][3606726432]a_terminate:1:current ref count = 1 2014-05-20 07:57:53.136: [ OCRAPI][3606726432]a_terminate:1:current ref count = 0 Check OLR configuration [root@grac41 Desktop]# ocrconfig -local -showbackup --> No backup [root@grac41 Desktop]# ocrcheck -local Status of Oracle Local Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 2704 Available space (kbytes) : 259416 ID : 1855884304 Device/File Name : /u01/app/11204/grid/cdata/grac41.olr Device/File integrity check succeeded Local registry integrity check succeeded Logical corruption check succeeded Dump OLR and display the failing resource: SYSTEM.GPnP.profiles.peer.pending [root@grac41 Desktop]# ocrdump -local --> [SYSTEM.GPnP] UNDEF : SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : grid, GROUP_NAME : oinstall} [SYSTEM.GPnP.profiles] BYTESTREAM (16) : SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : grid, GROUP_NAME : oinstall} [SYSTEM.GPnP.profiles.peer] UNDEF : SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : grid, GROUP_NAME : oinstall} [SYSTEM.GPnP.profiles.peer.best] SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : grid, GROUP_NAME : oinstall} [SYSTEM.GPnP.profiles.peer.pending] SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : root, GROUP_NAME : root} --> Why do we have root permissions for resource SYSTEM.GPnP.profiles.peer.pending] ? Maybe GPnP process was started as root and created above entry ? Anyway before spending to much time in debugging we need to restore OLR from backup or recreate OLR. Assuming we don't have a backup recreating OLR by following Note : 1193643.1 Recreating OLR by create an empty OLR file. # cd /u01/app/11204/grid/cdata/ # touch /u01/app/11204/grid/cdata/grac41.olr # chown grid:oinstall /u01/app/11204/grid/cdata/grac41.olr # chmod 600 /u01/app/11204/grid/cdata/grac41.olr # ls -l /u01/app/11204/grid/cdata/grac41.olr -rw-------. 1 grid oinstall 0 May 20 09:14 /u01/app/11204/grid/cdata/grac41.olr -> But doesn't work get PROCL-26 during clusterware restart Recreating OCR and OLR by following Doc ID 1539020.1 : # crsctl stop crs -f # $GRID_HOME/crs/install/rootcrs.pl -deconfig -force # $GRID_HOME/root.sh After less than 10 minutes CW is up and running again !
Reference:
- How to Recreate OLR When OLR Backup does not Exist or Corrupted (Doc ID 1539020.1)
- How to backup or restore OLR in 11.2 Grid Infrastructure (Doc ID 1193643.1)
In one of my case , activeversion entry was incorrect which caused cars not coming up. I have restored it from back then cars started. But customer wants to know why that corruption and what led it.