Troubleshooting Clusterware startup problems with DTRACE

Prepare DTRACE script check_rac.d  and run it first time

You need only change your grid location and the pid_file in the BEGIN section
grid_loc = "/u01/app/121/grid";    <<< Change this
pid_file="hract21.pid"             <<< Change this - replace hract21 with your hostname

As not all OS errors are fatal run DTRACE script  on a working node first : [ including comments ]
[root@hract21 DTRACE]# dtrace -s check_rac.d
dtrace: script 'check_rac.d' matched 21 probes
CPU     ID                    FUNCTION:NAME
0      1                           :BEGIN GRIDHOME: /u01/app/121/grid - GRIDHOME/bin: /u01/app/121/grid/bin  - Temp Loc: /var/tmp/.oracle -  PIDFILE: hract21.pid - Port for bind: 53
0      9                      open:return - Exec: ohasd.bin - open() /var/tmp/.oracle/npohasd failed with error: -6 - scan_dir:  /var/tmp/.oracle
0      9                      open:return - Exec: ohasd.bin - open() /var/tmp/.oracle/npohasd failed with error: -6 - scan_dir:  /var/tmp/.oracle
0      9                      open:return - Exec: ohasd.bin - open() /var/tmp/.oracle/npohasd failed with error: -6 - scan_dir:  /var/tmp/.oracle
0      9                      open:return - Exec: ohasd.bin - open() /var/tmp/.oracle/npohasd failed with error: -6 - scan_dir:  /var/tmp/.oracle
0      9                      open:return - Exec: ohasd.bin - open() /var/tmp/.oracle/npohasd failed with error: -6 - scan_dir:  /var/tmp/.oracle
---> You can ignore the     ENXIO 6    - No such device or address - during opening  /var/tmp/.oracle/npohasd
0     89                   connect:return - Exec: gipcd.bin - PID: 26471  connect() to Nameserver - fd : 27 - IP: 192.168.5.50 - Port: 53
0     89                   connect:return - Exec: gipcd.bin - PID: 26489  connect() to Nameserver - fd : 27 - IP: 192.168.5.50 - Port: 53
0     89                   connect:return - Exec: gipcd.bin - PID: 26505  connect() to Nameserver - fd : 27 - IP: 192.168.5.50 - Port: 53
---> gipcd.bin successfully connect to our name server
0     93                    sendto:return - Exec: mdnsd.bin - PID: 26428  sendto() failed with error : -32 - fd : 46
0     93                    sendto:return - Exec: crsd.bin - PID: 26874  sendto() failed with error : -32 - fd : 210
0    103                      bind:return - Exec: ons - PID: 27044  bind() failed with error : -98 - fd : 9 - IP: 0.0.0.0 - Port: 6200
0     89                   connect:return - Exec: gipcd.bin - PID: 26571  connect() failed with error : -113 - fd : 191 - IP: 192.168.5.123 - Port: 43956
0     89                   connect:return - Exec: gipcd.bin - PID: 26571  connect() failed with error : -113 - fd : 122 - IP: 192.168.5.123 - Port: 43956
0     89                   connect:return - Exec: gipcd.bin - PID: 26571  connect() failed with error : -113 - fd : 122 - IP: 192.168.5.123 - Port: 43956
0     89                   connect:return - Exec: gipcd.bin - PID: 26571  connect() failed with error : -113 - fd : 122 - IP: 192.168.5.123 - Port: 43956
0     89                   connect:return - Exec: gipcd.bin - PID: 26571  connect() failed with error : -113 - fd : 120 - IP: 192.168.5.123 - Port: 43956
0     89                   connect:return - Exec: gipcd.bin - PID: 26571  connect() failed with error : -113 - fd : 121 - IP: 192.168.5.123 - Port: 43956
0     89                   connect:return - Exec: gipcd.bin - PID: 26571  connect() failed with error : -113 - fd : 122 - IP: 192.168.5.123 - Port: 43956
0     89                   connect:return - Exec: gipcd.bin - PID: 26571  connect() failed with error : -113 - fd : 123 - IP: 192.168.5.123 - Port: 43956
--->  gipcd.bin fails connect to htrac23 node with error:     EHOSTUNREACH    113  No route to host -  as this node was down
0    123                    execve:return - Exec: oradism - execve()  /u01/app/121/grid/bin/oradism -  Lower CLUSTERWARE stack successfully started - ret code:  0 - EXITING !
---> As soon CW comes up / ASM run executable oradism we stop DTRACE script by calling exit !