Troubleshooting Clusterware and Clusterware component error : Address already in use

Generic RAC Portnumber Information

                                                  Default Port   Port Range  Protocol  Used for 
                                                  Number                               CI only? 
Cluster Synchronization Service daemon (CSSD)     42424          Dynamic     UDP       Yes
Oracle Grid Interprocess Communication (GIPCD)    42424          Dynamic     UDP       Yes
Oracle HA Services daemon (OHASD)                 42424          Dynamic     UDP       Yes
Multicast Domain Name Service (MDNSD)              5353          Dynamic     UDP/TCP    No 
Oracle Grid Naming Service (GNSD)                    53          53 (public) UDP/TCP    No
Oracle Notification Services (ONS)                 6100 (local)  Configured  TCP        No
                                                   6200 (remote)   manually
    
Port 42424 :
CSSD  : The Cluster Synchronization Service (CSS) daemon uses a fixed port for node restart 
        advisory messages.This port is used on all interfaces that have broadcast capability. 
        Broadcast  occurs only when a node  eviction restart is imminent.
OHASD : The Oracle High Availability Services (OHAS) daemon starts the Oracle Clusterware 
         stack.
GIPCD : A support daemon that enables Redundant Interconnect Usage.

Port 5353 :
MDNSD : The mDNS process is a background process on Linux and UNIX, and a service on Window, 
        and is necessary  for Grid Plug and Play and GNS.

Port 53: 
GNSD  : The Oracle Grid Naming Service daemon provides a gateway between the cluster mDNS and 
        external DNS servers. 
        The gnsd process performs name resolution within the cluster.

Port 6100/6200 :
ONS   : Port for ONS, used to publish and subscribe service for communicating information about 
        Fast Application Notification (FAN) events. The FAN notification process uses system 
        events that Oracle Database publishes  when cluster servers become unreachable or if 
        network interfaces fail.
        Use srvctl to modify ONS port

Verify port usage at OS level
As GNS runs only on a single node the cluster we need to Relocate GNS first :
[root@hract21 ~]# srvctl relocate gns -n hract21 

[root@hract21 Desktop]#  netstat -taupen |grep ":42424 "
udp        0      0 192.168.2.255:42424         0.0.0.0:*  0          10361774   11545/ohasd.bin     
udp        0      0 230.0.1.0:42424             0.0.0.0:*  0          10361773   11545/ohasd.bin     
udp        0      0 224.0.0.251:42424           0.0.0.0:*  0          10361772   11545/ohasd.bin     
udp        0      0 192.168.2.255:42424         0.0.0.0:*  501        10361732   11764/gipcd.bin     
udp        0      0 230.0.1.0:42424             0.0.0.0:*  501        10361731   11764/gipcd.bin     
udp        0      0 224.0.0.251:42424           0.0.0.0:*  501        10361730   11764/gipcd.bin     
udp        0      0 192.168.2.255:42424         0.0.0.0:*  501        10361722   11825/ocssd.bin     
udp        0      0 230.0.1.0:42424             0.0.0.0:*  501        10361721   11825/ocssd.bin     
udp        0      0 224.0.0.251:42424           0.0.0.0:*  501        10361720   11825/ocssd.bin 

[root@hract21 Desktop]# netstat -taupen |grep ":53 "
udp        0      0 192.168.5.58:53             0.0.0.0:*   0          46593880   5261/gnsd.bin  

[root@hract21 Desktop]#  netstat -taupen |grep ":5353 "
udp        0      0 0.0.0.0:5353            0.0.0.0:*  501        1378331    11724/mdnsd.bin     
udp        0      0 0.0.0.0:5353            0.0.0.0:*  501        1378210    11724/mdnsd.bin     
udp        0      0 0.0.0.0:5353            0.0.0.0:*  501        1378209    11724/mdnsd.bin     
udp        0      0 0.0.0.0:5353            0.0.0.0:*  501        1378208    11724/mdnsd.bin 

[root@hract21 Desktop]#  netstat -taupen |grep ":6100 "
tcp        0      0 127.0.0.1:6100     0.0.0.0:*     LISTEN  501  10419706   31762/ons    
..

 

Prepare Test program JavaUDPServer.java

Source can be found here : Simple Java example of UDP Client/Server communication

[root@hract21 JAVA]#  javac JavaUDPServer.java

Testing when a port is free and our program can successful listen to that port: 
[root@hract21 JAVA]# java  JavaUDPServer 59
Listening on UDP Port: 59
--> press <cntrl>C to terminate the program

Testing program  when port is already in use
[root@hract21 JAVA]# java  JavaUDPServer  53
Listening on UDP Port: 53
Jan 31, 2015 4:57:29 PM JavaUDPServer main
SEVERE: null
java.net.BindException: Address already in use
    at java.net.PlainDatagramSocketImpl.bind0(Native Method)
    at java.net.PlainDatagramSocketImpl.bind(PlainDatagramSocketImpl.java:125)
    at java.net.DatagramSocket.bind(DatagramSocket.java:372)

 

Case I: Clusterware startup fails as  Portnumber:  42424  is in use

Start our test program to block UPD port 42424
[root@hract21 JAVA]#  java  JavaUDPServer  42424
Listening on UDP Port: 42424

Start CRS and monitor local CRS stack
[root@hract21 Desktop]# crsct start crs
*****  Local Resources: *****
Resource NAME               INST   TARGET    STATE        SERVER          STATE_DETAILS
--------------------------- ----   ------------ ------------ --------------- -----------------------------------------
ora.asm                        1   ONLINE    OFFLINE      -               STABLE
ora.cluster_interconnect.haip  1   ONLINE    OFFLINE      -               STABLE
ora.crf                        1   ONLINE    OFFLINE      -               STABLE
ora.crsd                       1   ONLINE    OFFLINE      -               STABLE
ora.cssd                       1   ONLINE    OFFLINE      hract21         STARTING
ora.cssdmonitor                1   ONLINE     ONLINE       hract21         STABLE
ora.ctssd                      1   ONLINE    OFFLINE      -               STABLE
ora.diskmon                    1   OFFLINE    OFFLINE      -               STABLE
ora.drivers.acfs               1   ONLINE    ONLINE       hract21         STABLE
ora.evmd                       1   ONLINE    INTERMEDIATE hract21         STABLE
ora.gipcd                      1   ONLINE    ONLINE       hract21         STABLE
ora.gpnpd                      1   ONLINE    ONLINE       hract21         STABLE
ora.mdnsd                      1   ONLINE    ONLINE       hract21         STABLE
ora.storage                    1   ONLINE    OFFLINE      -               STABLE

--> evmd process remain in status INTERMEDIATE . Local cluster stack doesn't up !
Investigate Trace files:
alert.log: 
2015-01-31 17:14:54.492 [CSSDAGENT(22642)]CRS-5818: Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:9:3} in /u01/app/grid/diag/crs/hract21/crs/trace/ohasd_cssdagent_root.trc.
Sat Jan 31 17:14:59 2015
Errors in file /u01/app/grid/diag/crs/hract21/crs/trace/ocssd.trc  (incident=1):
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []

gipcd.trc:
2015-01-31 17:20:27.606277 :GIPCHTHR:812046080:  gipchaWorkerCreateInterface: created local interface for node 'hract21', haName 'gipcd_ha_name', inf 'udp://192.168.2.121:28764' inf 0x7fef0c190b30
2015-01-31 17:20:27.606350 :GIPCXCPT:812046080:  gipcmodNetworkProcessBind: failed to bind endp 0x7fef182d8230 [000000000001e71a] { gipcEndpoint : localAddr 'mcast://224.0.0.251:42424/192.168.2.121', remoteAddr '', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj (nil), sendp 0x7fef182da320 status 13flags 0x20000000, flags-2 0x0, usrFlags 0xc000 }, addr 0x7fef182d8cf0 [000000000001e71c] { gipcAddress : name 'mcast://224.0.0.251:42424/192.168.2.121', objFlags 0x0, addrFlags 0x5 }
2015-01-31 17:20:27.606358 :GIPCXCPT:812046080:  gipcmodNetworkProcessBind: slos op  :  sgipcnMctBind
2015-01-31 17:20:27.606360 :GIPCXCPT:812046080:  gipcmodNetworkProcessBind: slos dep :  Address already in use (98)
2015-01-31 17:20:27.606361 :GIPCXCPT:812046080:  gipcmodNetworkProcessBind: slos loc :  bind
2015-01-31 17:20:27.606363 :GIPCXCPT:812046080:  gipcmodNetworkProcessBind: slos info:  Invalid argument
2015-01-31 17:20:27.606399 :GIPCXCPT:812046080:  gipcBindF [gipcInternalEndpoint : gipcInternal.c : 468]: EXCEPTION[ ret gipcretAddressInUse (20) ]  failed to bind endp 0x7fef182d8230 [000000000001e71a] { gipcEndpoint : localAddr 'mcast://224.0.0.251:42424/192.168.2.121', remoteAddr '', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj (nil), sendp 0x7fef182da320 status 13flags 0x20000000, flags-2 0x0, usrFlags 0xc000 }, addr 0x7fef182d9a20 [000000000001e721] { gipcAddress : name 'mcast://224.0.0.251:42424/192.168.2.121', objFlags 0x0, addrFlags 0x4 }, flags 0x8000
2015-01-31 17:20:27.606408 :GIPCXCPT:812046080:  gipcInternalEndpoint: failed to bind address to endpoint name 'mcast://224.0.0.251:42424/192.168.2.121', ret gipcretAddressInUse (20)
2015-01-31 17:20:27.606426 :GIPCHTHR:812046080:  gipchaWorkerUpdateInterface: EXCEPTION[ ret gipcretAddressInUse (20) ]  failed to create local interface 'udp://192.168.2.121', 0x7fef0c190b30 { host '', haName 'gipcd_ha_name', local (nil), ip '192.168.2.121', subnet '192.168.2.0', mask '255.255.255.0', mac '08-00-27-4e-c9-bf', ifname 'eth2', numRef 0, numFail 0, idxBoot 0, flags 0x1841 }, hctx 0x10639b0 [0000000000000011] { gipchaContext : host 'hract21', name 'gipcd_ha_name', luid '8c45d6e7-00000000', name2 3aca-bf27-17d5-691e, numNode 0, numInf 1, maxPriority 0, clientMode 1, nodeIncarnation d64c9b7c-06451148 usrFlags 0x0, flags 0x2d65 }
2015-01-31 17:20:27.606432 :GIPCHGEN:812046080:  gipchaInterfaceDisable: disabling interface 0x7fef0c190b30 { host '', haName 'gipcd_ha_name', local (nil), ip '192.168.2.121', subnet '192.168.2.0', mask '255.255.255.0', mac '08-00-27-4e-c9-bf', ifname 'eth2', numRef 0, numFail 0, idxBoot 0, flags 0x1841 }
2015-01-31 17:20:27.606438 :GIPCHDEM:812046080:  gipchaWorkerCleanInterface: performing cleanup of disabled interface 0x7fef0c190b30 { host '', haName 'gipcd_ha_name', local (nil), ip '192.168.2.121', subnet '192.168.2.0', mask '255.255.255.0', mac '08-00-27-4e-c9-bf', ifname 'eth2', numRef 0, numFail 0, idxBoot 0, flags 0x1861 }
2015-01-31 17:20:27.60

Investigate the error more in detail  : 
gipcmodNetworkProcessBind: slos dep :  Address already in use (98) 
[root@hract21 Desktop]# cat  /usr/include/asm-generic/errno.h | grep 98
#define    EADDRINUSE    98    /* Address already in use */

Locate the port number :
gipcEndpoint : localAddr 'mcast://224.0.0.251:42424/192.168.2.121 -->   42424 is the port 
--> CW   can't listen on port 42424 ! 

Locate the  blocking process at OS level
[root@hract21 Desktop]#    netstat -taupen |grep ":42424 "
udp        0      0 :::42424         ....    22338/java          
[root@hract21 Desktop]# ps -elf | grep 22338
0 S root     22338 26783  0  80   0 - 438331 futex_ 17:04 pts/12  00:00:01 java JavaUDPServer 42424

--> Yep our java program blocks CW from comming up ! Kill java program and restart CW 
[root@hract21 Desktop]# kill -9 22338
[root@hract21 Desktop]# crsctl stop crs -f
[root@hract21 Desktop]# crsctl start crs

 

Case II: Clusterware startup fails as Portnumber  5353  is in use

Start our test program and block MDSND port 5353
[root@hract21 JAVA]#  java  JavaUDPServer 5353
Listening on UDP Port: 5353

*****  Local Resources: *****
Resource NAME               INST   TARGET       STATE        SERVER          STATE_DETAILS
--------------------------- ----   ------------ ------------ --------------- -----------------------------------------
ora.asm                        1   ONLINE       ONLINE       hract21         STABLE
ora.cluster_interconnect.haip  1   ONLINE    ONLINE       hract21         STABLE
ora.crf                        1   ONLINE    ONLINE       hract21         STABLE
ora.crsd                       1   ONLINE    ONLINE       hract21         STABLE
ora.cssd                       1   ONLINE    ONLINE       hract21         STABLE
ora.cssdmonitor                1   ONLINE     ONLINE       hract21         STABLE
ora.ctssd                      1   ONLINE    ONLINE       hract21         OBSERVER,STABLE
ora.diskmon                    1   OFFLINE    OFFLINE      -               STABLE
ora.drivers.acfs               1   ONLINE    ONLINE       hract21         STABLE
ora.evmd                       1   ONLINE    ONLINE       hract21         STABLE
ora.gipcd                      1   ONLINE    ONLINE       hract21         STABLE
ora.gpnpd                      1   ONLINE    ONLINE       hract21         STABLE
ora.mdnsd                      1   ONLINE    INTERMEDIATE hract21         STABLE
ora.storage                    1   ONLINE    ONLINE       hract21         STABLE
--> MDMSD daemon doesn' start 

mdnsd.trc :
Oracle Database 12c Clusterware Release 12.1.0.2.0 - Production Copyright 1996, 2014 Oracle. All rights reserved.
    CLSB:2559100480: Argument count (argc) for this daemon is 1
    CLSB:2559100480: Argument 0 is: /u01/app/121/grid/bin/mdnsd.bin
2015-01-31 17:40:17.131516 :  CLSDMT:2554820352: PID for the Process [9863], connkey 9
2015-01-31 17:40:18.042329 :    MDNS:2559100480:  mdnsd interface eth0 (0x2 AF=2 f=0x1043 mcast=-1) 192.168.1.9 mask 255.255.255.0 FAILED. Error 98 (Address already in use)
2015-01-31 17:40:18.043191 :    MDNS:2559100480:  mdnsd interface eth1 (0x3 AF=2 f=0x1043 mcast=-1) 192.168.5.121 mask 255.255.255.0 FAILED. Error 98 (Address already in use)
2015-01-31 17:40:18.046952 :    MDNS:2559100480:  mdnsd interface eth1:1 (0x3 AF=2 f=0x1043 mcast=-1) 192.168.5.241 mask 255.255.255.0 FAILED. Error 98 (Address already in use)
2015-01-31 17:40:18.047574 :    MDNS:2559100480:  mdnsd interface eth1:2 (0x3 AF=2 f=0x1043 mcast=-1) 192.168.5.242 mask 255.255.255.0 FAILED. Error 98 (Address already in use)
2015-01-31 17:40:18.047597 :    MDNS:2559100480:  mdnsd interface eth2 (0x4 AF=2 f=0x1043 mcast=-1) 192.168.2.121 mask 255.255.255.0 FAILED. Error 98 (Address already in use)
2015-01-31 17:40:18.047612 :    MDNS:2559100480:  mdnsd interface eth2:1 (0x4 AF=2 f=0x1043 mcast=-1) 169.254.213.86 mask 255.255.0.0 FAILED. Error 98 (Address already in use)
2015-01-31 17:40:18.049171 :    MDNS:2559100480:  mdnsd interface eth3 (0x5 AF=2 f=0x1043 mcast=-1) 192.168.3.121 mask 255.255.255.0 FAILED. Error 98 (Address already in use)
2015-01-31 17:40:18.049222 :    MDNS:2559100480:  mdnsd interface lo (0x1 AF=2 f=0x49 mcast=-1) 127.0.0.1 mask 255.0.0.0 FAILED. Error 98 (Address already in use)
2015-01-31 17:40:18.049236 :    MDNS:2559100480:  Error! No valid netowrk interfaces found to setup mDNS.
2015-01-31 17:40:18.049240 :    MDNS:2559100480:  Oracle mDNSResponder ver. mDNSResponder-1076 (Jun 30 2014 19:39:45) , init_rv=-65537
2015-01-31 17:40:18.049335 :    MDNS:2559100480:  stopping

--> Here we only get the error :  Address already in use  but info about  the portnumber. 
    We need to reference above list and remember that MSDNS is running on port 5353 

Now we can locate the blocking process , kill that process and restart clusterware
[root@hract21 Desktop]#  netstat -taupen |grep ":5353 "
udp        0      0 :::5353         ...            50111629   7252/java   
Again our java program prevents CW from startup. Kill the that process and resart CW.
[root@hract21 Desktop]# kill -9 7252

 

Case III: Investigate GNS startup problem due to Error:  Address already in use

Relocate GNS to a different host
[root@hract21 Desktop]# srvctl relocate gns -n hract23
ora.gns                        1   ONLINE       ONLINE       hract23         STABLE
ora.gns.vip                    1   ONLINE       ONLINE       hract23         STABLE

Now occupy port 53 by running our JAVA program:
[root@hract21 JAVA]# java  JavaUDPServer 53
Listening on UDP Port: 53

Now try to bring back the GNS 
[root@hract21 Desktop]# srvctl relocate gns -n hract21
*****  Cluster Resources: *****
Resource NAME               INST   TARGET       STATE        SERVER          STATE_DETAILS
--------------------------- ----   ------------ ------------ --------------- -----------------------------------------
ora.gns                        1   ONLINE       OFFLINE      hract21         STARTING
ora.gns.vip                    1   ONLINE       ONLINE       hract21         STABLE
--> GNS is in status STARTING but doesn't come up

gnsd.trc :
2015-01-31 18:09:13.518516 :GIPCXCPT:255158016:  gipcmodNetworkProcessBind: slos op  :  sgipcnTcpBind
2015-01-31 18:09:13.518518 :GIPCXCPT:255158016:  gipcmodNetworkProcessBind: slos dep :  Address already in use (98)
2015-01-31 18:09:13.518520 :GIPCXCPT:255158016:  gipcmodNetworkProcessBind: slos loc :  bind
2015-01-31 18:09:13.518521 :GIPCXCPT:255158016:  gipcmodNetworkProcessBind: slos info:  addr '192.168.5.58:53'
2015-01-31 18:09:13.518577 :GIPCXCPT:255158016:  gipcBindF [gipcInternalEndpoint : gipcInternal.c : 468]: EXCEPTION[ ret gipcretAddressInUse (20) ]  failed to bind endp 0x7ff7000034c0 [0000000000001fc6] { gipcEndpoint : localAddr 'udp://192.168.5.58:53', remoteAddr '', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj (nil), sendp 0x7ff7000050f0 status 13flags 0x20008000, flags-2 0x0, usrFlags 0x24000 }, addr 0x7ff7000047f0 [0000000000001fcd] { gipcAddress : name 'udp://192.168.5.58:53', objFlags 0x0, addrFlags 0x4 }, flags 0x20000
2015-01-31 18:09:13.518589 :GIPCXCPT:255158016:  gipcInternalEndpoint: failed to bind address to endpoint name 'udp://192.168.5.58:53', ret gipcretAddressInUse (20)
2015-01-31 18:09:13.518608 :GIPCXCPT:255158016:  gipcEndpointF [clsgngipcCreateEndpointInternal : clsgngipc.c : 2008]: EXCEPTION[ ret gipcretAddressInUse (20) ]  failed endp create ctx 0x7ff7196f3c80 [0000000000001e99] { gipcContext : traceLevel 2, fieldLevel 0x0, numDead 0, numPending 0, numZombie 0, numObj 4, numWait 0, numReady 0, wobj 0x7ff7196f1c10, hgid 0000000000001e9a, flags 0x1a, objFlags 0x0 }, name 'udp://192.168.5.58:53', flags 0x24000
2015-01-31 18:09:13.518728 :     GNS:255158016: Resolve::clsgndnsCreateContainerCallback: (:CLSGN01163:) Error - Address in use: port 53 address "192.168.5.58". 1: clskec:has:CLSGN:208 2 args[has:CLSGN:208][udp://192.168.5.58:53]
2: clskec:has:gipc:20 1 args[has:gipc:20]
3: clskec:has:CLSU:910 4 args[has][mod=gipcInternalEndpoint][loc=473][msg=failed to bind address to endpoint name 'udp://192.168.5.58:53']
2015-01-31 18:09:13.518769 :     GNS:255158016: Resolve::clsgndnsCreateContainer: (:CLSGN00927:) failed to listen on all addresses - throwing error.
default:255158016: listen failed with 1 errors
1: clskec:has:CLSGN:208 3 args[has:CLSGN:208][192.168.5.58][53]

The following error messages tell use Linux errno code and the related portnumber :
2015-01-31 18:09:13.518518 :GIPCXCPT:255158016:  gipcmodNetworkProcessBind: slos dep :  Address already in use (98)
2015-01-31 18:09:13.518521 :GIPCXCPT:255158016:  gipcmodNetworkProcessBind: slos info:  addr '192.168.5.58:53'

Again locate the port number and kill the process
[root@hract21 Desktop]#  netstat -taupen |grep ":53 "
udp    16128      0 :::53          ...          51417680   23723/java
Again kill the process which holds the port number and restart CW 
[root@hract21 Desktop]# kill -9  23723

Now test whether Relocated GNS works again
[root@hract21 ~]#   srvctl relocate gns -n hract21
*****  Cluster Resources: *****
Resource NAME               INST   TARGET       STATE        SERVER          STATE_DETAILS
--------------------------- ----   ------------ ------------ --------------- -----------------------------------------
ora.gns                        1   ONLINE       ONLINE       hract21         STABLE
ora.gns.vip                    1   ONLINE       ONLINE       hract21         STABL

Complete Portnumber Usage of a working RAC system

[root@hract21 ~]#   netstat -taupen |grep 192.168
tcp        0      0 192.168.5.242:1521          0.0.0.0:*                   LISTEN      501        50803141   17310/tnslsnr       
tcp        0      0 192.168.5.241:1521          0.0.0.0:*                   LISTEN      501        50793916   17258/tnslsnr       
tcp        0      0 192.168.5.121:1521          0.0.0.0:*                   LISTEN      501        50793894   17258/tnslsnr       
tcp        0      0 192.168.2.121:1522          0.0.0.0:*                   LISTEN      501        50790436   17212/tnslsnr       
tcp        0      0 192.168.2.121:61020         0.0.0.0:*                   LISTEN      0          50773311   16994/osysmond.bin  
tcp        0      0 192.168.5.121:42942         0.0.0.0:*                   LISTEN      501        50724207   16454/gipcd.bin     
tcp        0      0 192.168.5.58:39839          0.0.0.0:*                   LISTEN      0          51856456   27381/gnsd.bin      
tcp        0      0 192.168.5.232:36063         0.0.0.0:*                   LISTEN      502        8145376    596/exectask        
tcp        0      0 192.168.5.121:15043         0.0.0.0:*                   LISTEN      0          50730332   16281/ohasd.bin     
tcp        0      0 192.168.5.121:42942         192.168.5.123:28657         ESTABLISHED 501        50841598   16454/gipcd.bin     
tcp        0      0 192.168.5.241:1521          192.168.5.121:55119         ESTABLISHED 501        50829166   17258/tnslsnr       
tcp        0      0 192.168.2.121:46847         192.168.2.122:1522          ESTABLISHED 0          50774509   17012/crsd.bin      
tcp        0      0 192.168.2.121:1522          192.168.2.123:60331         ESTABLISHED 501        50795614   17212/tnslsnr       
tcp        0      0 192.168.2.121:1522          192.168.2.121:16025         ESTABLISHED 501        50829535   17212/tnslsnr       
tcp        0      0 192.168.2.121:1522          192.168.2.122:54611         ESTABLISHED 501        50796842   17212/tnslsnr       
tcp        0      0 192.168.2.121:46865         192.168.2.122:1522          ESTABLISHED 501        50829527   17468/asm_lreg_+ASM 
tcp        0      0 192.168.5.242:1521          192.168.5.121:61101         ESTABLISHED 501        50838159   17310/tnslsnr       
tcp        0      0 192.168.5.121:42942         192.168.5.122:32304         ESTABLISHED 501        50841582   16454/gipcd.bin     
tcp        1      0 192.168.1.9:39471           80.150.192.73:80            CLOSE_WAIT  0          50900520   4786/clock-applet   
tcp        0      0 192.168.2.121:1522          192.168.2.121:16024         ESTABLISHED 501        50829534   17212/tnslsnr       
tcp        0      0 192.168.2.121:16024         192.168.2.121:1522          ESTABLISHED 501        50829529   17468/asm_lreg_+ASM 
tcp        0      0 192.168.2.121:16025         192.168.2.121:1522          ESTABLISHED 501        50829531   17468/asm_lreg_+ASM 
tcp        0      0 192.168.2.121:28139         192.168.2.123:1522          ESTABLISHED 501        50829525   17468/asm_lreg_+ASM 
tcp        0      0 192.168.5.121:64227         192.168.5.122:35547         ESTABLISHED 501        50790718   16454/gipcd.bin     
tcp        0      0 192.168.5.121:21046         192.168.5.123:6200          ESTABLISHED 501        50787900   17215/ons           
tcp        0      0 192.168.5.121:59844         192.168.5.50:22             ESTABLISHED 0          44509382   13726/ssh           
tcp        0      0 192.168.5.121:61101         192.168.5.242:1521          ESTABLISHED 502        50838158   17721/ora_lreg_bank 
tcp        0      0 192.168.5.58:39839          192.168.5.121:34266         TIME_WAIT   0          0          -                   
tcp        0      0 192.168.5.121:16432         192.168.5.122:6200          ESTABLISHED 501        50787901   17215/ons           
tcp        0      0 192.168.2.121:39861         192.168.2.123:61021         ESTABLISHED 0          50769440   16994/osysmond.bin  
tcp        0      0 192.168.5.121:55125         192.168.5.241:1521          ESTABLISHED 502        50837652   17721/ora_lreg_bank 
tcp        0      0 192.168.5.121:55119         192.168.5.241:1521          ESTABLISHED 501        50829165   17468/asm_lreg_+ASM 
tcp        0      0 192.168.5.241:1521          192.168.5.121:55125         ESTABLISHED 501        50837653   17258/tnslsnr       
tcp        0      0 192.168.5.121:10242         192.168.5.123:17701         ESTABLISHED 501        50790723   16454/gipcd.bin     
tcp        0      0 192.168.5.121:55728         192.168.5.123:22            ESTABLISHED 0          27679552   25184/ssh           
udp        0      0 192.168.2.121:35570         0.0.0.0:*                               0          50731287   16281/ohasd.bin     
udp        0      0 192.168.2.121:51962         0.0.0.0:*                               0          50751183   16922/octssd.bin    
udp        0      0 192.168.2.255:42424         0.0.0.0:*                               501        50734962   16537/ocssd.bin     
udp        0      0 192.168.2.255:42424         0.0.0.0:*                               0          50731290   16281/ohasd.bin     
udp        0      0 192.168.2.255:42424         0.0.0.0:*                               501        50725223   16454/gipcd.bin     
udp        0      0 192.168.2.121:15891         0.0.0.0:*                               501        50734959   16537/ocssd.bin     
udp        0      0 192.168.2.121:12075         0.0.0.0:*                               501        50782505   16408/evmd.bin      
udp        0      0 192.168.5.58:53             0.0.0.0:*                               0          51856599   27381/gnsd.bin      
udp        0      0 192.168.5.58:123            0.0.0.0:*                               38         51843931   1291/ntpd           
udp        0      0 192.168.5.242:123           0.0.0.0:*                               38         50803109   1291/ntpd           
udp        0      0 192.168.5.241:123           0.0.0.0:*                               38         50793859   1291/ntpd           
udp        0      0 192.168.3.121:123           0.0.0.0:*                               0          43573989   1291/ntpd           
udp        0      0 192.168.2.121:123           0.0.0.0:*                               0          43573987   1291/ntpd           
udp        0      0 192.168.5.121:123           0.0.0.0:*                               0          43573984   1291/ntpd           
udp        0      0 192.168.1.9:123             0.0.0.0:*                               0          43573983   1291/ntpd           
udp        0      0 192.168.2.121:53498         0.0.0.0:*                               0          50776026   17012/crsd.bin      
udp        0      0 192.168.2.121:45379         0.0.0.0:*                               501        50725220   16454/gipcd.bin

Troubleshooting hint CW startproblems due to  Address already in use errors

Before CW startup verify the the following ports are not in use at all 
[root@hract21 Desktop]#    netstat -taupen |grep ":42424 "
[root@hract21 Desktop]#    netstat -taupen |grep ":5353 "
[root@hract21 Desktop]#    netstat -taupen |grep ":53 "
[root@hract21 Desktop]#    netstat -taupen |egrep ":6100 |:6200"
If you find any processes not belonging to the Oracle Clusterware stack you need to kill/stop 
these processes


If having problem with Clusterware startup or CW components startup ( GSD, VIPs ) you may 
check your clusterware tracefils for  "Address already in use" Error .

Note the tracefile location has changed for RAC 12.1.0.2 :
[grid@hract21 trace]$  grep -l "Address already in use" *
gipcd.trc
gnsd.trc
mdnsd.trc
ocssd.trc
ohasd.trc

Now find details:
# grep "Address already in use" ohasd.trc  mdnsd.trc  ocssd.trc gnsd.trc  gipcd.trc gnsd.trc | grep "2015-01-31 17"
ohasd.trc:2015-01-31 17:30:16.613432 :GIPCXCPT:2420897536:  gipcmodNetworkProcessBind: slos dep :  Address already in use (98)
mdnsd.trc:2015-01-31 17:40:18.049222 :    MDNS:2559100480:  mdnsd interface lo (0x1 AF=2 f=0x49 mcast=-1) 127.0.0.1 mask 255.0.0.0 FAILED. Error 98 (Address already in use)
ocssd.trc:2015-01-31 17:04:56.013085 :GIPCXCPT:3986515712:  gipcmodNetworkProcessBind: slos dep :  Address already in use (98)
gipcd.trc:2015-01-31 17:06:25.204775 :GIPCXCPT:812046080:   gipcmodNetworkProcessBind: slos dep :  Address already in use (98)
gnsd.trc:2015-01-31 18:09:13.518518 :GIPCXCPT:2551580oblematic Portnumber and 16:    gipcmodNetworkProcessBind: slos dep :  Address already in use (98)

For mdnsd.trc we already know the port number          :   5353   
For ohasd.trc, ocssd.trc, gipcd.trc the port number is :  42424
For GNS the tracefiles provides details about the problematic Portnumber and IP-adress
[grid@hract21 trace]$  grep gipcmodNetworkProcessBind  gnsd.trc  
2015-01-31 18:09:13.518483 :GIPCXCPT:255158016:  gipcmodNetworkProcessBind: failed to bind endp 0x7ff7000034c0 [0000000000001fc6] { gipcEndpoint : localAddr 'udp://192.168.5.58:53', remoteAddr '', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj (nil), sendp 0x7ff7000050f0 status 13flags 0x20008000, flags-2 0x0, usrFlags 0x24000 }, addr 0x7ff7000038b0 [0000000000001fc8] { gipcAddress : name 'udp://192.168.5.58:53', objFlags 0x0, addrFlags 0x5 }
2015-01-31 18:09:13.518516 :GIPCXCPT:255158016:  gipcmodNetworkProcessBind: slos op  :  sgipcnTcpBind
2015-01-31 18:09:13.518518 :GIPCXCPT:255158016:  gipcmodNetworkProcessBind: slos dep :  Address already in use (98)
2015-01-31 18:09:13.518520 :GIPCXCPT:255158016:  gipcmodNetworkProcessBind: slos loc :  bind
2015-01-31 18:09:13.518521 :GIPCXCPT:255158016:  gipcmodNetworkProcessBind: slos info:  addr '192.168.5.58:53'

 

Reference

Leave a Reply

Your email address will not be published. Required fields are marked *