Recreate GNS 12102

Backup profile.xml and OCR and gather data of current GNS setup

As of 12.1/11.2 Grid Infrastructure, the private network configuration is not only stored in OCR but also in the 
gpnp profile -  please take a backup of profile.xml on all cluster nodes before proceeding, as grid user:

[root@hract21 ~]# cd $GRID_HOME/gpnp/hract21/profiles/peer/
[root@hract21 peer]#  cp profile.xml profile.xml_bk-2-FEB-2015
[root@hract21 peer]#  ocrconfig -local -manualbackup
hract21     2015/02/02 09:04:23     /u01/app/121/grid/cdata/hract21/backup_20150202_090423.olr     0     
hract21     2015/01/30 12:40:51     /u01/app/121/grid/cdata/hract21/backup_20150130_124051.olr     0     
[root@hract21 peer]#  ocrconfig -local -showbackup
hract21     2015/02/02 09:04:23     /u01/app/121/grid/cdata/hract21/backup_20150202_090423.olr     0     
hract21     2015/01/30 12:40:51     /u01/app/121/grid/cdata/hract21/backup_20150130_124051.olr     0  

[root@hract21 peer]# oifcfg getif
eth1  192.168.5.0  global  public
eth2  192.168.2.0  global  cluster_interconnect,asm

[root@hract21 peer]# crsctl status resource ora.gns.vip -f | grep USR_ORA_VIP
USR_ORA_VIP=192.168.5.58

[root@hract21 peer]#  ifconfig eth1 | egrep 'eth|inet addr'
eth1      Link encap:Ethernet  HWaddr 08:00:27:7D:8E:49  
          inet addr:192.168.5.121  Bcast:192.168.5.255  Mask:255.255.255.0
[root@hract21 peer]# ifconfig eth2  | egrep 'eth|inet addr'
eth2      Link encap:Ethernet  HWaddr 08:00:27:4E:C9:BF  
          inet addr:192.168.2.121  Bcast:192.168.2.255  Mask:255.255.255.0
[root@hract21 peer]#  ifconfig eth3   | egrep 'eth|inet addr'
eth3      Link encap:Ethernet  HWaddr 08:00:27:3B:89:BF  
          inet addr:192.168.3.121  Bcast:192.168.3.255  Mask:255.255.255.0

[root@hract21 peer]#  srvctl config gns -a -l
GNS is enabled.
GNS is listening for DNS server requests on port 53
GNS is using port 5353 to connect to mDNS
GNS status: OK
Domain served by GNS: grid12c.example.com
GNS version: 12.1.0.2.0
Globally unique identifier of the cluster where GNS is running: 3d7c30fc9a0eeff3ff12b79970a14c12
Name of the cluster where GNS is running: ract2
Cluster type: server.
GNS log level: 1.
GNS listening addresses: tcp://192.168.5.58:39839.
GNS is individually enabled on nodes: 
GNS is individually disabled on nodes: 

Stop resources and recreate  gns, nodeapps

[root@hract21 peer]#  srvctl stop scan_listener 
[root@hract21 peer]#  srvctl stop scan
[root@hract21 peer]#  srvctl stop nodeapps -f
PRCC-1016 : ons was already stopped
PRCR-1005 : Resource ora.ons is already stopped

[root@hract21 peer]#  srvctl stop gns
[root@hract21 Desktop]#  srvctl remove gns 
Remove GNS? (y/[n]) y


[root@hract21 Desktop]# srvctl remove nodeapps
Please confirm that you intend to remove node-level applications on all nodes of the cluster (y/[n]) y
[root@hract21 Desktop]# srvctl  add gns -i 192.168.5.58 -d  grid12c.example.com
[root@hract21 Desktop]# srvctl config gns
GNS is enabled.
GNS VIP addresses: 192.168.5.58
Domain served by GNS: grid12c.example.com
[root@hract21 Desktop]# srvctl config gns -list
CLSNS-00005: operation timed out
  CLSNS-00025: unable to locate GNS
    CLSGN-00070: Service location failed.
[root@hract21 Desktop]# srvctl start gns
[root@hract21 Desktop]# srvctl config gns -list
Oracle-GNS A 192.168.5.58 Unique Flags: 0x115
ract2.Oracle-GNS SRV Target: Oracle-GNS Protocol: tcp Port: 46453 Weight: 0 Priority: 0 Flags: 0x115
ract2.Oracle-GNS TXT CLUSTER_NAME="ract2", CLUSTER_GUID="3d7c30fc9a0eeff3ff12b79970a14c12", NODE_NAME="hract21", SERVER_STATE="RUNNING", VERSION="12.1.0.2.0", DOMAIN="grid12c.example.com" Flags: 0x115
--> No VIPs there  

Recreate Nodeapps
[root@hract21 Desktop]#  srvctl add nodeapps -S 192.168.5.0/255.255.255.0/eth1 
 [root@hract21 Desktop]#  srvctl config gns -list
Oracle-GNS A 192.168.5.58 Unique Flags: 0x115
hract21-vip A 192.168.5.246 Unique Flags: 0x1
hract22-vip A 192.168.5.239 Unique Flags: 0x1
hract23-vip A 192.168.5.244 Unique Flags: 0x1
ract2.Oracle-GNS SRV Target: Oracle-GNS Protocol: tcp Port: 46453 Weight: 0 Priority: 0 Flags: 0x115
ract2.Oracle-GNS TXT CLUSTER_NAME="ract2", CLUSTER_GUID="3d7c30fc9a0eeff3ff12b79970a14c12", NODE_NAME="hract21", SERVER_STATE="RUNNING", VERSION="12.1.0.2.0", DOMAIN="grid12c.example.com" Flags: 0x115
--> Now VIPs should be ONLINE 
*****  Cluster Resources: *****
Resource NAME               INST   TARGET       STATE        SERVER          STATE_DETAILS
--------------------------- ----   ------------ ------------ --------------- -----------------------------------------
ora.hract21.vip                1   ONLINE       ONLINE       hract21         STABLE  
ora.hract22.vip                1   ONLINE       ONLINE       hract22         STABLE  
ora.hract23.vip                1   ONLINE       ONLINE       hract23         STABLE 

Restart SCAN and SCAN Listeners
[root@hract21 Desktop]#  srvctl start scan
--> Now SCANs should be ONLINE
ora.scan1.vip                  1   ONLINE       ONLINE       hract22         STABLE  
ora.scan2.vip                  1   ONLINE       ONLINE       hract23         STABLE  
ora.scan3.vip                  1   ONLINE       ONLINE       hract21         STABLE  

[root@hract21 Desktop]# srvctl start scan_listener
--> Now SCAN_LISTENER should be ONLINE
*****  Cluster Resources: *****
Resource NAME               INST   TARGET       STATE        SERVER          STATE_DETAILS
--------------------------- ----   ------------ ------------ --------------- -----------------------------------------
ora.LISTENER_SCAN1.lsnr        1   ONLINE       ONLINE       hract22         STABLE  
ora.LISTENER_SCAN2.lsnr        1   ONLINE       ONLINE       hract23         STABLE  
ora.LISTENER_SCAN3.lsnr        1   ONLINE       ONLINE       hract21         STABLE  

Verify GNS
[root@hract21 Desktop]#   srvctl config gns -list
Oracle-GNS A 192.168.5.58 Unique Flags: 0x115
hract21-vip A 192.168.5.246 Unique Flags: 0x1
hract22-vip A 192.168.5.239 Unique Flags: 0x1
hract23-vip A 192.168.5.244 Unique Flags: 0x1
ract2-scan A 192.168.5.238 Unique Flags: 0x1
ract2-scan A 192.168.5.243 Unique Flags: 0x1
ract2-scan A 192.168.5.245 Unique Flags: 0x1
ract2-scan1-vip A 192.168.5.243 Unique Flags: 0x1
ract2-scan2-vip A 192.168.5.245 Unique Flags: 0x1
ract2-scan3-vip A 192.168.5.238 Unique Flags: 0x1
ract2.Oracle-GNS SRV Target: Oracle-GNS Protocol: tcp Port: 46453 Weight: 0 Priority: 0 Flags: 0x115
ract2.Oracle-GNS TXT CLUSTER_NAME="ract2", CLUSTER_GUID="3d7c30fc9a0eeff3ff12b79970a14c12", NODE_NAME="hract21", 
   SERVER_STATE="RUNNING", VERSION="12.1.0.2.0", DOMAIN="grid12c.example.com" Flags: 0x115
--> VIPS, SCAN and SCAN VIPS should be ONLINE 
    Congrats you have successfully reconfigured GNS on 12.1.0.2 !

Potential problem : PRCN-2065,PRCN-2067  during recreating nodeapps

Note stopping nodeapps should stop the ONS !
[grid@hract21 trace]$  srvctl stop nodeapps -n hract21 -f
*****  Local Resources: *****
Rescource NAME                 TARGET     STATE           SERVER       STATE_DETAILS                       
-------------------------      ---------- ----------      ------------ ------------------                  
ora.ons                        OFFLINE    OFFLINE         hract21      STABLE   
ora.ons                        ONLINE     ONLINE          hract22      STABLE   
ora.ons                        ONLINE     ONLINE          hract23      STABLE   
[root@hract21 Desktop]# netstat -tapen | egrep '6100|6200'
-> Ons is stopped - port 6100 and 6200 not actice !
Sometimes during my testing  the remote  ONS port was still active after  :
  srvctl stop nodeapps -f
Later if we try to create the nodeapps we get the following error:
[root@hract21 Desktop]#  srvctl add nodeapps -S 192.168.5.0/255.255.255.0/eth1
PRCN-2065 : Ports 6200 are not available on the nodes given
PRCN-2067 : Port 6200 is not available on nodes: hract21,hract22,hract23

Verify TCP prot  status :
[root@hract22 ~]# netstat -taupen | grep 6200
tcp        0      0 :::6200                    ..  LISTEN      501        441704     21856/ons           
tcp        0      0 ::ffff:192.168.5.122:6200  ..  ESTABLISHED 501        67450915   21856/ons           
tcp        0      0 ::ffff:192.168.5.122:6200  ..  ESTABLISHED 501        72457163   21856/ons 
ONS was still running a occupied port 6200. This creates the above error ! 

WA: use the -skip parameter ( for details please read BUG 18317414 ) 
What is this really doing ?
[root@hract21 Desktop]# srvctl add nodeapps -skip -help
    -skip        Skip reachability check of VIP address and port validation for ONS

Now recreate the nodeapps with the skip paramter
[root@hract21 Desktop]#   srvctl add nodeapps  -skip  -S 192.168.5.0/255.255.255.0/eth1
--> Worked !!

Reference

  • Bug 18317414 : LNX64-12.1-INSTALL-SCC:RERUN ROOT.SH FAILED AT ADD NODEAPPS

One thought on “Recreate GNS 12102”

  1. I’ve learn a few good stuff here. Certainly worth bookmarking for revisiting.
    I surprise how much effort you put to create such a fantastic
    informative web site.

Leave a Reply

Your email address will not be published. Required fields are marked *