Backup profile.xml and OCR and gather data of current GNS setup
As of 12.1/11.2 Grid Infrastructure, the private network configuration is not only stored in OCR but also in the
gpnp profile - please take a backup of profile.xml on all cluster nodes before proceeding, as grid user:
[root@hract21 ~]# cd $GRID_HOME/gpnp/hract21/profiles/peer/
[root@hract21 peer]# cp profile.xml profile.xml_bk-2-FEB-2015
[root@hract21 peer]# ocrconfig -local -manualbackup
hract21 2015/02/02 09:04:23 /u01/app/121/grid/cdata/hract21/backup_20150202_090423.olr 0
hract21 2015/01/30 12:40:51 /u01/app/121/grid/cdata/hract21/backup_20150130_124051.olr 0
[root@hract21 peer]# ocrconfig -local -showbackup
hract21 2015/02/02 09:04:23 /u01/app/121/grid/cdata/hract21/backup_20150202_090423.olr 0
hract21 2015/01/30 12:40:51 /u01/app/121/grid/cdata/hract21/backup_20150130_124051.olr 0
[root@hract21 peer]# oifcfg getif
eth1 192.168.5.0 global public
eth2 192.168.2.0 global cluster_interconnect,asm
[root@hract21 peer]# crsctl status resource ora.gns.vip -f | grep USR_ORA_VIP
USR_ORA_VIP=192.168.5.58
[root@hract21 peer]# ifconfig eth1 | egrep 'eth|inet addr'
eth1 Link encap:Ethernet HWaddr 08:00:27:7D:8E:49
inet addr:192.168.5.121 Bcast:192.168.5.255 Mask:255.255.255.0
[root@hract21 peer]# ifconfig eth2 | egrep 'eth|inet addr'
eth2 Link encap:Ethernet HWaddr 08:00:27:4E:C9:BF
inet addr:192.168.2.121 Bcast:192.168.2.255 Mask:255.255.255.0
[root@hract21 peer]# ifconfig eth3 | egrep 'eth|inet addr'
eth3 Link encap:Ethernet HWaddr 08:00:27:3B:89:BF
inet addr:192.168.3.121 Bcast:192.168.3.255 Mask:255.255.255.0
[root@hract21 peer]# srvctl config gns -a -l
GNS is enabled.
GNS is listening for DNS server requests on port 53
GNS is using port 5353 to connect to mDNS
GNS status: OK
Domain served by GNS: grid12c.example.com
GNS version: 12.1.0.2.0
Globally unique identifier of the cluster where GNS is running: 3d7c30fc9a0eeff3ff12b79970a14c12
Name of the cluster where GNS is running: ract2
Cluster type: server.
GNS log level: 1.
GNS listening addresses: tcp://192.168.5.58:39839.
GNS is individually enabled on nodes:
GNS is individually disabled on nodes:
Stop resources and recreate gns, nodeapps
[root@hract21 peer]# srvctl stop scan_listener
[root@hract21 peer]# srvctl stop scan
[root@hract21 peer]# srvctl stop nodeapps -f
PRCC-1016 : ons was already stopped
PRCR-1005 : Resource ora.ons is already stopped
[root@hract21 peer]# srvctl stop gns
[root@hract21 Desktop]# srvctl remove gns
Remove GNS? (y/[n]) y
[root@hract21 Desktop]# srvctl remove nodeapps
Please confirm that you intend to remove node-level applications on all nodes of the cluster (y/[n]) y
[root@hract21 Desktop]# srvctl add gns -i 192.168.5.58 -d grid12c.example.com
[root@hract21 Desktop]# srvctl config gns
GNS is enabled.
GNS VIP addresses: 192.168.5.58
Domain served by GNS: grid12c.example.com
[root@hract21 Desktop]# srvctl config gns -list
CLSNS-00005: operation timed out
CLSNS-00025: unable to locate GNS
CLSGN-00070: Service location failed.
[root@hract21 Desktop]# srvctl start gns
[root@hract21 Desktop]# srvctl config gns -list
Oracle-GNS A 192.168.5.58 Unique Flags: 0x115
ract2.Oracle-GNS SRV Target: Oracle-GNS Protocol: tcp Port: 46453 Weight: 0 Priority: 0 Flags: 0x115
ract2.Oracle-GNS TXT CLUSTER_NAME="ract2", CLUSTER_GUID="3d7c30fc9a0eeff3ff12b79970a14c12", NODE_NAME="hract21", SERVER_STATE="RUNNING", VERSION="12.1.0.2.0", DOMAIN="grid12c.example.com" Flags: 0x115
--> No VIPs there
Recreate Nodeapps
[root@hract21 Desktop]# srvctl add nodeapps -S 192.168.5.0/255.255.255.0/eth1
[root@hract21 Desktop]# srvctl config gns -list
Oracle-GNS A 192.168.5.58 Unique Flags: 0x115
hract21-vip A 192.168.5.246 Unique Flags: 0x1
hract22-vip A 192.168.5.239 Unique Flags: 0x1
hract23-vip A 192.168.5.244 Unique Flags: 0x1
ract2.Oracle-GNS SRV Target: Oracle-GNS Protocol: tcp Port: 46453 Weight: 0 Priority: 0 Flags: 0x115
ract2.Oracle-GNS TXT CLUSTER_NAME="ract2", CLUSTER_GUID="3d7c30fc9a0eeff3ff12b79970a14c12", NODE_NAME="hract21", SERVER_STATE="RUNNING", VERSION="12.1.0.2.0", DOMAIN="grid12c.example.com" Flags: 0x115
--> Now VIPs should be ONLINE
***** Cluster Resources: *****
Resource NAME INST TARGET STATE SERVER STATE_DETAILS
--------------------------- ---- ------------ ------------ --------------- -----------------------------------------
ora.hract21.vip 1 ONLINE ONLINE hract21 STABLE
ora.hract22.vip 1 ONLINE ONLINE hract22 STABLE
ora.hract23.vip 1 ONLINE ONLINE hract23 STABLE
Restart SCAN and SCAN Listeners
[root@hract21 Desktop]# srvctl start scan
--> Now SCANs should be ONLINE
ora.scan1.vip 1 ONLINE ONLINE hract22 STABLE
ora.scan2.vip 1 ONLINE ONLINE hract23 STABLE
ora.scan3.vip 1 ONLINE ONLINE hract21 STABLE
[root@hract21 Desktop]# srvctl start scan_listener
--> Now SCAN_LISTENER should be ONLINE
***** Cluster Resources: *****
Resource NAME INST TARGET STATE SERVER STATE_DETAILS
--------------------------- ---- ------------ ------------ --------------- -----------------------------------------
ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE hract22 STABLE
ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE hract23 STABLE
ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE hract21 STABLE
Verify GNS
[root@hract21 Desktop]# srvctl config gns -list
Oracle-GNS A 192.168.5.58 Unique Flags: 0x115
hract21-vip A 192.168.5.246 Unique Flags: 0x1
hract22-vip A 192.168.5.239 Unique Flags: 0x1
hract23-vip A 192.168.5.244 Unique Flags: 0x1
ract2-scan A 192.168.5.238 Unique Flags: 0x1
ract2-scan A 192.168.5.243 Unique Flags: 0x1
ract2-scan A 192.168.5.245 Unique Flags: 0x1
ract2-scan1-vip A 192.168.5.243 Unique Flags: 0x1
ract2-scan2-vip A 192.168.5.245 Unique Flags: 0x1
ract2-scan3-vip A 192.168.5.238 Unique Flags: 0x1
ract2.Oracle-GNS SRV Target: Oracle-GNS Protocol: tcp Port: 46453 Weight: 0 Priority: 0 Flags: 0x115
ract2.Oracle-GNS TXT CLUSTER_NAME="ract2", CLUSTER_GUID="3d7c30fc9a0eeff3ff12b79970a14c12", NODE_NAME="hract21",
SERVER_STATE="RUNNING", VERSION="12.1.0.2.0", DOMAIN="grid12c.example.com" Flags: 0x115
--> VIPS, SCAN and SCAN VIPS should be ONLINE
Congrats you have successfully reconfigured GNS on 12.1.0.2 !
Potential problem : PRCN-2065,PRCN-2067 during recreating nodeapps
Note stopping nodeapps should stop the ONS !
[grid@hract21 trace]$ srvctl stop nodeapps -n hract21 -f
***** Local Resources: *****
Rescource NAME TARGET STATE SERVER STATE_DETAILS
------------------------- ---------- ---------- ------------ ------------------
ora.ons OFFLINE OFFLINE hract21 STABLE
ora.ons ONLINE ONLINE hract22 STABLE
ora.ons ONLINE ONLINE hract23 STABLE
[root@hract21 Desktop]# netstat -tapen | egrep '6100|6200'
-> Ons is stopped - port 6100 and 6200 not actice !
Sometimes during my testing the remote ONS port was still active after :
srvctl stop nodeapps -f
Later if we try to create the nodeapps we get the following error:
[root@hract21 Desktop]# srvctl add nodeapps -S 192.168.5.0/255.255.255.0/eth1
PRCN-2065 : Ports 6200 are not available on the nodes given
PRCN-2067 : Port 6200 is not available on nodes: hract21,hract22,hract23
Verify TCP prot status :
[root@hract22 ~]# netstat -taupen | grep 6200
tcp 0 0 :::6200 .. LISTEN 501 441704 21856/ons
tcp 0 0 ::ffff:192.168.5.122:6200 .. ESTABLISHED 501 67450915 21856/ons
tcp 0 0 ::ffff:192.168.5.122:6200 .. ESTABLISHED 501 72457163 21856/ons
ONS was still running a occupied port 6200. This creates the above error !
WA: use the -skip parameter ( for details please read BUG 18317414 )
What is this really doing ?
[root@hract21 Desktop]# srvctl add nodeapps -skip -help
-skip Skip reachability check of VIP address and port validation for ONS
Now recreate the nodeapps with the skip paramter
[root@hract21 Desktop]# srvctl add nodeapps -skip -S 192.168.5.0/255.255.255.0/eth1
--> Worked !!
Reference
- Bug 18317414 : LNX64-12.1-INSTALL-SCC:RERUN ROOT.SH FAILED AT ADD NODEAPPS
I’ve learn a few good stuff here. Certainly worth bookmarking for revisiting.
I surprise how much effort you put to create such a fantastic
informative web site.