Troubleshooting Clusterware startup problems with DTRACE

Debug Clusterware startup problems with DTRACE

Why to use DTRACE ?

  • Using DTRACE scripts can save you many hours of debugging
  • DTRACE scritps can be used to capture intermittent errors
  • DTRACE can easily configured to track very specific things like:   [ How many bytes where written by the 5.th I/O for file XXXX after it gets newly created  ]
  • DTRACE can provide very detail information about system call usage and timing
  • DTRACE is the next level of debugging on Linux much better than strace
  • Note :  Not  all reported errors by DTRACE  are significant and prevents CW from booting
  • Best practise is to test the start on a working node to figure out what is relevant or not

If you change a lot of things for testing and if you have many RAC instances running
  where lot of people are testing  DTRACE will be become you best friend !

 

Version used and OS error codes handled by DTRACE script

  • OEL 6.6 with Linux hract21.example.com 3.8.13-55.1.2.el6uek.x86_64
  • Oracle 12.1.0.2
Installed Dtrace packages:
[root@hract21 Desktop]# rpm -qa | grep dtrace
dtrace-utils-0.4.5-2.el6.x86_64
dtrace-utils-devel-0.4.5-2.el6.x86_64
libdtrace-ctf-0.4.1-1.x86_64
dtrace-modules-3.8.13-44.1.1.el6uek-0.4.3-4.el6.x86_64
libdtrace-ctf-devel-0.4.1-1.x86_64
dtrace-modules-headers-0.4.3-4.el6.x86_64
dtrace-modules-3.8.13-55.1.2.el6uek-0.4.3-4.el6.x86_64

Error Codes tracked by dtrace script: 
/usr/include/asm-generic/errno-base.h

#define    ENOENT        2    /* No such file or directory */
#define    ENXIO         6    /* No such device or address */
#define    EAGAIN        11    /* Try again */
#define    EACCES        13    /* Permission denied */

/usr/include/asm-generic/errno.h
#define    EADDRINUSE       98    /* Address already in use */
#define    EADDRNOTAVAIL    99    /* Cannot assign requested address */
#define    ECONNREFUSED    111    /* Connection refused */
#define    EHOSTUNREACH    113    /* No route to host */
#define    EINPROGRESS     115    /* Operation now in progress */

2 thoughts on “Troubleshooting Clusterware startup problems with DTRACE”

Leave a Reply

Your email address will not be published. Required fields are marked *