Backup current OLR and all available OLR backups – don’t miss this step
# ocrconfig -local -manualbackup
grac41 2014/05/20 12:22:20 /u01/app/11204/grid/cdata/grac41/backup_20140520_122220.olr
Verify available backups
# ocrconfig -local -showbackup
grac41 2014/05/20 12:22:20 /u01/app/11204/grid/cdata/grac41/backup_20140520_122220.olr
..
# ls -l /u01/app/11204/grid/cdata/grac41/*
-rw-------. 1 root root 6803456 Sep 12 2013 /u01/app/11204/grid/cdata/grac41/backup_20130912_130457.olr
-rw-------. 1 root root 7016448 May 20 09:08 /u01/app/11204/grid/cdata/grac41/backup_20140520_090842.olr
-rw-------. 1 root root 6807552 May 20 12:22 /u01/app/11204/grid/cdata/grac41/backup_20140520_122220.olr
Restore a OLR from backup
Shutdown CRS and check that ohasd.bin process is not running and restore OLR
# crsctl stop crs
# ps -elf | egrep "PID|d.bin|ohas|oraagent|orarootagent|cssdagent|cssdmonitor" | grep -v grep
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
4 S root 4494 1 0 80 0 - 2846 wait 10:04 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run
# ocrconfig -local -restore /u01/app/11204/grid/cdata/grac41/backup_20140520_090842.olr
# crsctl stop crs
Recover OLR after PROCL-26. PROCL-5 error and no OLR backups available
Tracefile:
# crsctl start crs
TraceFileName: ./client/ocrcheck_27844.log
2014-05-20 09:38:48.918: [OCRCHECK][2768283392]Failed to access OCR repository. Retval [26] Error [PROCL-26: Error while accessing the physical storage]
TraceFileName: ./gpnpd/gpnpd_1.log
2014-05-19 13:23:19.100: [ GPNP][85341984]procr_open_key_ext: OLR error message : PROCL-5: User does not have permission to perform a local registry operation on this key. Authentication error
[User does not have permission to perform this operation] [0]
Details of PROCL-5 from ./gpnpd/gpnpd_1.log
..
[ CLWAL][3606726432]clsw_Initialize: OLR initlevel [70000]
[ OCRAPI][3606726432]a_init:10: AUTH LOC [/u01/app/11204/grid/srvm/auth]
[ OCRMSG][3606726432]prom_init: Successfully registered comp [OCRMSG] in clsd.
2014-05-20 07:57:51.118: [ OCRAPI][3606726432]a_init:11: Messaging init successful.
[ OCRCLI][3606726432]oac_init: Successfully registered comp [OCRCLI] in clsd.
2014-05-20 07:57:51.118: [ OCRCLI][3606726432]proac_con_init: Local listener using IPC. [(ADDRESS=(PROTOCOL=ipc)(KEY=procr_local_conn_0_PROL))]
2014-05-20 07:57:51.119: [ OCRCLI][3606726432]proac_con_init: Successfully connected to the server
2014-05-20 07:57:51.119: [ OCRCLI][3606726432]proac_con_init: Post sema. Con count [1]
2014-05-20 07:57:51.120: [ OCRAPI][3606726432]a_init:12: Client init successful.
2014-05-20 07:57:51.120: [ OCRAPI][3606726432]a_init:21: OCR init successful. Init Level [7]
2014-05-20 07:57:51.120: [ OCRAPI][3606726432]a_init:2: Init Level [7]
2014-05-20 07:57:51.132: [ OCRCLI][3606726432]proac_con_init: Post sema. Con count [2]
[ clsdmt][3595089664]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=grac41DBG_GPNPD))
2014-05-20 07:57:51.133: [ clsdmt][3595089664]PID for the Process [31034], connkey 10
2014-05-20 07:57:51.133: [ clsdmt][3595089664]Creating PID [31034] file for home /u01/app/11204/grid host grac41 bin gpnp to /u01/app/11204/grid/gpnp/init/
2014-05-20 07:57:51.133: [ clsdmt][3595089664]Writing PID [31034] to the file [/u01/app/11204/grid/gpnp/init/grac41.pid]
2014-05-20 07:57:52.108: [ GPNP][3606726432]clsgpnpd_validateProfile: [at clsgpnpd.c:2919] GPnPD taken cluster name 'grac4'
2014-05-20 07:57:52.108: [ GPNP][3606726432]clsgpnpd_openLocalProfile: [at clsgpnpd.c:3477] Got local profile from file cache provider (LCP-FS).
2014-05-20 07:57:52.111: [ GPNP][3606726432]clsgpnpd_openLocalProfile: [at clsgpnpd.c:3532] Got local profile from OLR cache provider (LCP-OLR).
2014-05-20 07:57:52.113: [ GPNP][3606726432]procr_open_key_ext: OLR api procr_open_key_ext failed for key SYSTEM.GPnP.profiles.peer.pending
2014-05-20 07:57:52.113: [ GPNP][3606726432]procr_open_key_ext: OLR current boot level : 7
2014-05-20 07:57:52.113: [ GPNP][3606726432]procr_open_key_ext: OLR error code : 5
2014-05-20 07:57:52.126: [ GPNP][3606726432]procr_open_key_ext: OLR error message : PROCL-5: User does not have permission to perform a local registry operation on this key. Authentication error
[User does not have permission to perform this operation] [0]
2014-05-20 07:57:52.126: [ GPNP][3606726432]clsgpnpco_ocr2profile: [at clsgpnpco.c:578] Result: (58) CLSGPNP_OCR_ERR. Failed to open requested OLR Profile.
2014-05-20 07:57:52.127: [ GPNP][3606726432]clsgpnpd_lOpen: [at clsgpnpd.c:1734] Listening on ipc://GPNPD_grac41
2014-05-20 07:57:52.127: [ GPNP][3606726432]clsgpnpd_lOpen: [at clsgpnpd.c:1743] GIPC gipcretFail (1) gipcListen listen failure on
2014-05-20 07:57:52.127: [ default][3606726432]GPNPD failed to start listening for GPnP peers.
2014-05-20 07:57:52.135: [ GPNP][3606726432]clsgpnpd_term: [at clsgpnpd.c:1344] STOP GPnPD terminating. Closing connections...
2014-05-20 07:57:52.137: [ default][3606726432]clsgpnpd_term STOP terminating.
2014-05-20 07:57:53.136: [ OCRAPI][3606726432]a_terminate:1:current ref count = 1
2014-05-20 07:57:53.136: [ OCRAPI][3606726432]a_terminate:1:current ref count = 0
Check OLR configuration
[root@grac41 Desktop]# ocrconfig -local -showbackup
--> No backup
[root@grac41 Desktop]# ocrcheck -local
Status of Oracle Local Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2704
Available space (kbytes) : 259416
ID : 1855884304
Device/File Name : /u01/app/11204/grid/cdata/grac41.olr
Device/File integrity check succeeded
Local registry integrity check succeeded
Logical corruption check succeeded
Dump OLR and display the failing resource: SYSTEM.GPnP.profiles.peer.pending
[root@grac41 Desktop]# ocrdump -local
-->
[SYSTEM.GPnP]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : grid, GROUP_NAME : oinstall}
[SYSTEM.GPnP.profiles]
BYTESTREAM (16) :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : grid, GROUP_NAME : oinstall}
[SYSTEM.GPnP.profiles.peer]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : grid, GROUP_NAME : oinstall}
[SYSTEM.GPnP.profiles.peer.best]
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : grid, GROUP_NAME : oinstall}
[SYSTEM.GPnP.profiles.peer.pending]
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : root, GROUP_NAME : root}
--> Why do we have root permissions for resource SYSTEM.GPnP.profiles.peer.pending] ?
Maybe GPnP process was started as root and created above entry ?
Anyway before spending to much time in debugging we need to restore OLR from backup or recreate OLR.
Assuming we don't have a backup recreating OLR by following Note : 1193643.1
Recreating OLR by create an empty OLR file.
# cd /u01/app/11204/grid/cdata/
# touch /u01/app/11204/grid/cdata/grac41.olr
# chown grid:oinstall /u01/app/11204/grid/cdata/grac41.olr
# chmod 600 /u01/app/11204/grid/cdata/grac41.olr
# ls -l /u01/app/11204/grid/cdata/grac41.olr
-rw-------. 1 grid oinstall 0 May 20 09:14 /u01/app/11204/grid/cdata/grac41.olr
-> But doesn't work get PROCL-26 during clusterware restart
Recreating OCR and OLR by following Doc ID 1539020.1 :
# crsctl stop crs -f
# $GRID_HOME/crs/install/rootcrs.pl -deconfig -force
# $GRID_HOME/root.sh
After less than 10 minutes CW is up and running again !
Reference:
- How to Recreate OLR When OLR Backup does not Exist or Corrupted (Doc ID 1539020.1)
- How to backup or restore OLR in 11.2 Grid Infrastructure (Doc ID 1193643.1)
In one of my case , activeversion entry was incorrect which caused cars not coming up. I have restored it from back then cars started. But customer wants to know why that corruption and what led it.
Hello Kirankumar,
How are you ? I left Bangalore in Oct 2006 and searched you a lot in social media but did not find. If you see this ….Please Email me on mksocp@gmail.com
Regards,
Manoranjan Singh