Backup profile.xml and OCR and gather data of current GNS setup
As of 12.1/11.2 Grid Infrastructure, the private network configuration is not only stored in OCR but also in the gpnp profile - please take a backup of profile.xml on all cluster nodes before proceeding, as grid user: [root@hract21 ~]# cd $GRID_HOME/gpnp/hract21/profiles/peer/ [root@hract21 peer]# cp profile.xml profile.xml_bk-2-FEB-2015 [root@hract21 peer]# ocrconfig -local -manualbackup hract21 2015/02/02 09:04:23 /u01/app/121/grid/cdata/hract21/backup_20150202_090423.olr 0 hract21 2015/01/30 12:40:51 /u01/app/121/grid/cdata/hract21/backup_20150130_124051.olr 0 [root@hract21 peer]# ocrconfig -local -showbackup hract21 2015/02/02 09:04:23 /u01/app/121/grid/cdata/hract21/backup_20150202_090423.olr 0 hract21 2015/01/30 12:40:51 /u01/app/121/grid/cdata/hract21/backup_20150130_124051.olr 0 [root@hract21 peer]# oifcfg getif eth1 192.168.5.0 global public eth2 192.168.2.0 global cluster_interconnect,asm [root@hract21 peer]# crsctl status resource ora.gns.vip -f | grep USR_ORA_VIP USR_ORA_VIP=192.168.5.58 [root@hract21 peer]# ifconfig eth1 | egrep 'eth|inet addr' eth1 Link encap:Ethernet HWaddr 08:00:27:7D:8E:49 inet addr:192.168.5.121 Bcast:192.168.5.255 Mask:255.255.255.0 [root@hract21 peer]# ifconfig eth2 | egrep 'eth|inet addr' eth2 Link encap:Ethernet HWaddr 08:00:27:4E:C9:BF inet addr:192.168.2.121 Bcast:192.168.2.255 Mask:255.255.255.0 [root@hract21 peer]# ifconfig eth3 | egrep 'eth|inet addr' eth3 Link encap:Ethernet HWaddr 08:00:27:3B:89:BF inet addr:192.168.3.121 Bcast:192.168.3.255 Mask:255.255.255.0 [root@hract21 peer]# srvctl config gns -a -l GNS is enabled. GNS is listening for DNS server requests on port 53 GNS is using port 5353 to connect to mDNS GNS status: OK Domain served by GNS: grid12c.example.com GNS version: 12.1.0.2.0 Globally unique identifier of the cluster where GNS is running: 3d7c30fc9a0eeff3ff12b79970a14c12 Name of the cluster where GNS is running: ract2 Cluster type: server. GNS log level: 1. GNS listening addresses: tcp://192.168.5.58:39839. GNS is individually enabled on nodes: GNS is individually disabled on nodes:
Stop resources and recreate gns, nodeapps
[root@hract21 peer]# srvctl stop scan_listener [root@hract21 peer]# srvctl stop scan [root@hract21 peer]# srvctl stop nodeapps -f PRCC-1016 : ons was already stopped PRCR-1005 : Resource ora.ons is already stopped [root@hract21 peer]# srvctl stop gns [root@hract21 Desktop]# srvctl remove gns Remove GNS? (y/[n]) y [root@hract21 Desktop]# srvctl remove nodeapps Please confirm that you intend to remove node-level applications on all nodes of the cluster (y/[n]) y [root@hract21 Desktop]# srvctl add gns -i 192.168.5.58 -d grid12c.example.com [root@hract21 Desktop]# srvctl config gns GNS is enabled. GNS VIP addresses: 192.168.5.58 Domain served by GNS: grid12c.example.com [root@hract21 Desktop]# srvctl config gns -list CLSNS-00005: operation timed out CLSNS-00025: unable to locate GNS CLSGN-00070: Service location failed. [root@hract21 Desktop]# srvctl start gns [root@hract21 Desktop]# srvctl config gns -list Oracle-GNS A 192.168.5.58 Unique Flags: 0x115 ract2.Oracle-GNS SRV Target: Oracle-GNS Protocol: tcp Port: 46453 Weight: 0 Priority: 0 Flags: 0x115 ract2.Oracle-GNS TXT CLUSTER_NAME="ract2", CLUSTER_GUID="3d7c30fc9a0eeff3ff12b79970a14c12", NODE_NAME="hract21", SERVER_STATE="RUNNING", VERSION="12.1.0.2.0", DOMAIN="grid12c.example.com" Flags: 0x115 --> No VIPs there Recreate Nodeapps [root@hract21 Desktop]# srvctl add nodeapps -S 192.168.5.0/255.255.255.0/eth1 [root@hract21 Desktop]# srvctl config gns -list Oracle-GNS A 192.168.5.58 Unique Flags: 0x115 hract21-vip A 192.168.5.246 Unique Flags: 0x1 hract22-vip A 192.168.5.239 Unique Flags: 0x1 hract23-vip A 192.168.5.244 Unique Flags: 0x1 ract2.Oracle-GNS SRV Target: Oracle-GNS Protocol: tcp Port: 46453 Weight: 0 Priority: 0 Flags: 0x115 ract2.Oracle-GNS TXT CLUSTER_NAME="ract2", CLUSTER_GUID="3d7c30fc9a0eeff3ff12b79970a14c12", NODE_NAME="hract21", SERVER_STATE="RUNNING", VERSION="12.1.0.2.0", DOMAIN="grid12c.example.com" Flags: 0x115 --> Now VIPs should be ONLINE ***** Cluster Resources: ***** Resource NAME INST TARGET STATE SERVER STATE_DETAILS --------------------------- ---- ------------ ------------ --------------- ----------------------------------------- ora.hract21.vip 1 ONLINE ONLINE hract21 STABLE ora.hract22.vip 1 ONLINE ONLINE hract22 STABLE ora.hract23.vip 1 ONLINE ONLINE hract23 STABLE Restart SCAN and SCAN Listeners [root@hract21 Desktop]# srvctl start scan --> Now SCANs should be ONLINE ora.scan1.vip 1 ONLINE ONLINE hract22 STABLE ora.scan2.vip 1 ONLINE ONLINE hract23 STABLE ora.scan3.vip 1 ONLINE ONLINE hract21 STABLE [root@hract21 Desktop]# srvctl start scan_listener --> Now SCAN_LISTENER should be ONLINE ***** Cluster Resources: ***** Resource NAME INST TARGET STATE SERVER STATE_DETAILS --------------------------- ---- ------------ ------------ --------------- ----------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE hract22 STABLE ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE hract23 STABLE ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE hract21 STABLE Verify GNS [root@hract21 Desktop]# srvctl config gns -list Oracle-GNS A 192.168.5.58 Unique Flags: 0x115 hract21-vip A 192.168.5.246 Unique Flags: 0x1 hract22-vip A 192.168.5.239 Unique Flags: 0x1 hract23-vip A 192.168.5.244 Unique Flags: 0x1 ract2-scan A 192.168.5.238 Unique Flags: 0x1 ract2-scan A 192.168.5.243 Unique Flags: 0x1 ract2-scan A 192.168.5.245 Unique Flags: 0x1 ract2-scan1-vip A 192.168.5.243 Unique Flags: 0x1 ract2-scan2-vip A 192.168.5.245 Unique Flags: 0x1 ract2-scan3-vip A 192.168.5.238 Unique Flags: 0x1 ract2.Oracle-GNS SRV Target: Oracle-GNS Protocol: tcp Port: 46453 Weight: 0 Priority: 0 Flags: 0x115 ract2.Oracle-GNS TXT CLUSTER_NAME="ract2", CLUSTER_GUID="3d7c30fc9a0eeff3ff12b79970a14c12", NODE_NAME="hract21", SERVER_STATE="RUNNING", VERSION="12.1.0.2.0", DOMAIN="grid12c.example.com" Flags: 0x115 --> VIPS, SCAN and SCAN VIPS should be ONLINE Congrats you have successfully reconfigured GNS on 12.1.0.2 !
Potential problem : PRCN-2065,PRCN-2067 during recreating nodeapps
Note stopping nodeapps should stop the ONS ! [grid@hract21 trace]$ srvctl stop nodeapps -n hract21 -f ***** Local Resources: ***** Rescource NAME TARGET STATE SERVER STATE_DETAILS ------------------------- ---------- ---------- ------------ ------------------ ora.ons OFFLINE OFFLINE hract21 STABLE ora.ons ONLINE ONLINE hract22 STABLE ora.ons ONLINE ONLINE hract23 STABLE [root@hract21 Desktop]# netstat -tapen | egrep '6100|6200' -> Ons is stopped - port 6100 and 6200 not actice ! Sometimes during my testing the remote ONS port was still active after : srvctl stop nodeapps -f Later if we try to create the nodeapps we get the following error: [root@hract21 Desktop]# srvctl add nodeapps -S 192.168.5.0/255.255.255.0/eth1 PRCN-2065 : Ports 6200 are not available on the nodes given PRCN-2067 : Port 6200 is not available on nodes: hract21,hract22,hract23 Verify TCP prot status : [root@hract22 ~]# netstat -taupen | grep 6200 tcp 0 0 :::6200 .. LISTEN 501 441704 21856/ons tcp 0 0 ::ffff:192.168.5.122:6200 .. ESTABLISHED 501 67450915 21856/ons tcp 0 0 ::ffff:192.168.5.122:6200 .. ESTABLISHED 501 72457163 21856/ons ONS was still running a occupied port 6200. This creates the above error ! WA: use the -skip parameter ( for details please read BUG 18317414 ) What is this really doing ? [root@hract21 Desktop]# srvctl add nodeapps -skip -help -skip Skip reachability check of VIP address and port validation for ONS Now recreate the nodeapps with the skip paramter [root@hract21 Desktop]# srvctl add nodeapps -skip -S 192.168.5.0/255.255.255.0/eth1 --> Worked !!
Reference
- Bug 18317414 : LNX64-12.1-INSTALL-SCC:RERUN ROOT.SH FAILED AT ADD NODEAPPS
I’ve learn a few good stuff here. Certainly worth bookmarking for revisiting.
I surprise how much effort you put to create such a fantastic
informative web site.