Trace File Analyzer Collector – TFA Collector Usage and installation

Best TFA practices

  • To cleanup TFA BDB you need to uninstall/install TFA
  • Always use at least 4h for diag collection period ( 4h is default )
  • Check TFA repository for free space
  • Add your OSWatcher files  by running : tfactl directory add /home/OSW/oswbb/archive
  • Always provide graphical output for OSWatcher data for each node
  •  Sample:   # java -jar oswbba.jar -i  ./new_archive  -P grac1 -6 -7 -8  -B  Oct 23 10:00:00 2013 -E Oct 23 12:00:00 2013
  • Check that zip file for all nodes are created – rerun trace collection command if not
# ls /u01/app/grid/tfa/repository/collection_Wed_Oct_23_12_26_11_CEST_2013_node_all/*.zip
/u01/app/grid/tfa/repository/collection_Wed_Oct_23_12_26_11_CEST_2013_node_all/grac1.tfa_Wed_Oct_23_12_26_11_CEST_2013.zip
/u01/app/grid/tfa/repository/collection_Wed_Oct_23_12_26_11_CEST_2013_node_all/grac2.tfa_Wed_Oct_23_12_26_11_CEST_2013.zip
/u01/app/grid/tfa/repository/collection_Wed_Oct_23_12_26_11_CEST_2013_node_all/grac3.tfa_Wed_Oct_23_12_26_11_CEST_2013.zip
  • Check above zip files are holding major trace files ( rdbms alert.log / CRS alert.log / ocssd.log )
# unzip -l grac1.tfa_Wed_Oct_23_12_26_11_CEST_2013.zip | egrep 'alert|ocssd.log'
  2019952  10-23-2013 12:26   grac1//u01/app/11203/grid/log/grac1/cssd/ocssd.log
    10802  10-23-2013 12:27   grac1/rdbms/grace2/grac21/trace/alert_grac21.log
      353  10-23-2013 12:27   grac1//u01/app/11203/grid/log/grac1/alertgrac1.log

Reinstall TFA ( > 11.2.0.4 ) on a specific node

Check status 
[root@grac42 ~]# $GRID_HOME/bin/tfactl print status
.----------------------------------------------------------------------------------------------.
| Host   | Status of TFA | PID   | Port | Version    | Build ID             | Inventory Status |
+--------+---------------+-------+------+------------+----------------------+------------------+
| grac42 | RUNNING       |  1015 | 5000 | 12.1.2.0.0 | 12120020140619094932 | COMPLETE         |
| grac41 | RUNNING       | 31453 | 5000 | 12.1.2.0.0 | 12120020140619094932 | COMPLETE         |
'--------+---------------+-------+------+------------+----------------------+------------------'

Uninstal TFA on grac42
[root@grac42 ~]#  $GRID_HOME/bin/tfactl  uninstall
TFA will be Uninstalled on Node grac42: 
Removing TFA from grac42 only
Please remove TFA locally on any other configured nodes
.
[root@grac41 Desktop]#  $GRID_HOME/bin/tfactl print status
.----------------------------------------------------------------------------------------------.
| Host   | Status of TFA | PID   | Port | Version    | Build ID             | Inventory Status |
+--------+---------------+-------+------+------------+----------------------+------------------+
| grac41 | RUNNING       | 31453 | 5000 | 12.1.2.0.0 | 12120020140619094932 | COMPLETE         |
'--------+---------------+-------+------+------------+----------------------+------------------'
--> TFA removed on node grac42

Re-install TFA 
[root@grac42 ~]# $GRID_HOME/crs/install/tfa_setup.sh -silent -crshome /u01/app/11204/grid
Starting TFA installation
Using JAVA_HOME : /u01/app/11204/grid/jdk/jre
..
Installing TFA on grac42:
HOST: grac42    TFA_HOME: /u01/app/11204/grid/tfa/grac42/tfa_home

.--------------------------------------------------------------------------.
| Host   | Status of TFA | PID  | Port | Version    | Build ID             |
+--------+---------------+------+------+------------+----------------------+
| grac42 | RUNNING       | 8457 | 5000 | 12.1.2.0.0 | 12120020140619094932 |
'--------+---------------+------+------+------------+----------------------'

Verify TFA status
root@grac42 ~]#    $GRID_HOME/bin/tfactl print status
.----------------------------------------------------------------------------------------------.
| Host   | Status of TFA | PID   | Port | Version    | Build ID             | Inventory Status |
+--------+---------------+-------+------+------------+----------------------+------------------+
| grac42 | RUNNING       |  8457 | 5000 | 12.1.2.0.0 | 12120020140619094932 | COMPLETE         |
| grac41 | RUNNING       | 31453 | 5000 | 12.1.2.0.0 | 12120020140619094932 | COMPLETE         |
'--------+---------------+-------+------+------------+----------------------+------------------'
--> TFA reinstalled on node grac42

Installation of TFA Collector on top of 11.2.0.3

#  ./installTFALite.sh
Starting TFA installation
Enter a location for installing TFA (/tfa will be appended if not supplied) [/home/oracle/RAC/TFA]: 
  /u01/app/grid
Enter a Java Home that contains Java 1.5 or later : 
  /usr/lib/jvm/jre-1.6.0-openjdk.x86_64
Would you like to do a [L]ocal only or [C]lusterwide installation ? [L|l|C|c] [C] : C
TFA Will be Installed on the Following Nodes:
++++++++++++++++++++++++++++++++++++++++++++
Install Nodes
=============
grac1
grac2
grac3
Do you wish to make changes to the Node List ? [Y/y/N/n] [N] N
Installing TFA on grac1
HOST: grac1    TFA_HOME: /u01/app/grid/tfa/grac1/tfa_home
Installing TFA on grac2
HOST: grac2    TFA_HOME: /u01/app/grid/tfa/grac2/tfa_home
Installing TFA on grac3
HOST: grac3    TFA_HOME: /u01/app/grid/tfa/grac3/tfa_home
Host grac2 is part of TFA cluster
Host grac3 is part of TFA cluster
.-----------------------------------------------.
| Host  | Status of TFA | PID  | Port | Version |
+-------+---------------+------+------+---------+
| grac1 | RUNNING       | 9241 | 5000 | 2.5.1.5 |
| grac2 | RUNNING       | 6462 | 5000 | 2.5.1.5 |
| grac3 | RUNNING       | 6555 | 5000 | 2.5.1.5 |
'-------+---------------+------+------+---------'
Summary of TFA Installation:
.--------------------------------------------------------.
|                          grac1                         |
+---------------------+----------------------------------+
| Parameter           | Value                            |
+---------------------+----------------------------------+
| Install location    | /u01/app/grid/tfa/grac1/tfa_home |
| Repository location | /u01/app/grid/tfa/repository     |
| Repository usage    | 0 MB out of 2629 MB              |
'---------------------+----------------------------------'
.--------------------------------------------------------.
|                          grac2                         |
+---------------------+----------------------------------+
| Parameter           | Value                            |
+---------------------+----------------------------------+
| Install location    | /u01/app/grid/tfa/grac2/tfa_home |
| Repository location | /u01/app/grid/tfa/repository     |
| Repository usage    | 0 MB out of 2629 MB              |
'---------------------+----------------------------------'
.--------------------------------------------------------.
|                          grac3                         |
+---------------------+----------------------------------+
| Parameter           | Value                            |
+---------------------+----------------------------------+
| Install location    | /u01/app/grid/tfa/grac3/tfa_home |
| Repository location | /u01/app/grid/tfa/repository     |
| Repository usage    | 0 MB out of 2629 MB              |
'---------------------+----------------------------------'
Removed ssh equivalency setup on grac3
Removed ssh equivalency setup on grac2
TFA is successfully installed..

Start , stop and shutdown TFA

Stop TFA
# /etc/init.d/init.tfa stop
Stopping TFA
TFA is running  - Will wait 5 seconds (up to 3 times)  
TFA is running  - Will wait 5 seconds (up to 3 times)  
TFA is running  - Will wait 5 seconds (up to 3 times)  
TFAmain Force Stopped Successfully
. . . 
Successfully stopped TFA..

Start TFA
# /etc/init.d/init.tfa  start
Starting TFA..
start: Job is already running: oracle-tfa
Waiting up to 100 seconds for TFA to be started..
. . . . . 
.
. . . . . 
Successfully started TFA Process..
. . . . . 
TFA Started and listening for commands

Other useful  commands 
Restart TFA
# /etc/init.d/init.tfa  restart

Stop TFAMain process and removes related inittab entries
# /etc/init.d/init.tfa shutdown

Verify TFA status

Verify TFA runtime status
# /u01/app/grid/tfa/bin/tfactl print status
.-----------------------------------------------.
| Host  | Status of TFA | PID  | Port | Version |
+-------+---------------+------+------+---------+
| grac1 | RUNNING       | 9241 | 5000 | 2.5.1.5 |
| grac2 | RUNNING       | 6462 | 5000 | 2.5.1.5 |
| grac3 | RUNNING       | 6555 | 5000 | 2.5.1.5 |
'-------+---------------+------+------+---------'

Show current TFA configuration

# /u01/app/grid/tfa/bin/tfactl print config
.---------------------------------------------------.
| Configuration Parameter                 | Value   |
+-----------------------------------------+---------+
| TFA version                             | 2.5.1.5 |
| Automatic diagnostic collection         | OFF     |
| Trimming of files during diagcollection | ON      |
| Repository current size (MB) in grac1   | 6       |
| Repository maximum size (MB) in grac1   | 2629    |
| Trace level                             | 1       |
'-----------------------------------------+---------'

Use tfactl to check for out standing actions ( for example during diacollect )

#   $GRID_HOME/tfa/bin/tfactl  print  actions
.------------------------------------------------------------------------.
| HOST   | TIME         | ACTION         | STATUS  | COMMENTS            |
+--------+--------------+----------------+---------+---------------------+
| grac42 | Mar 04 08:56 | Collect traces | RUNNING | Collection details: |
|        |              | & zip          |         |                     |
|        |              |                |         | Zip file:           |
|        |              |                |         | tfa_Tue_Mar_4_08_56 |
|        |              |                |         | _25_CET_2014.zip    |
|        |              |                |         | Tag:                |
|        |              |                |         | collection_Tue_Mar_ |
|        |              |                |         | 4_08_56_25_CET_2014 |
|        |              |                |         | _node_all           |
+--------+--------------+----------------+---------+---------------------+
| grac42 | Mar 04 08:56 | Run inventory  | RUNNING | RDBMS               |
|        |              |                |         | all:ASM:CRS:DBWLM:A |
|        |              |                |         | CFS:CRS:ASM:OS:INST |
|        |              |                |         | ALL:TNS:CHMOS       |
+--------+--------------+----------------+---------+---------------------+
| grac41 | Mar 04 08:56 | Run inventory  | RUNNING | -c:RDBMS            |
|        |              |                |         | all:ASM:CRS:DBWLM:A |
|        |              |                |         | CFS:CRS:ASM:OS:INST |
|        |              |                |         | ALL:TNS:CHMOS       |
+--------+--------------+----------------+---------+---------------------+
| grac41 | Mar 04 08:56 | Collect traces | RUNNING | Collection details: |
|        |              | & zip          |         |                     |
|        |              |                |         | Zip file:           |
|        |              |                |         | tfa_Tue_Mar_4_08_56 |
|        |              |                |         | _25_CET_2014.zip    |
|        |              |                |         | Tag:                |
|        |              |                |         | collection_Tue_Mar_ |
|        |              |                |         | 4_08_56_25_CET_2014 |
|        |              |                |         | _node_all           |
+--------+--------------+----------------+---------+---------------------+
| grac43 | Mar 04 08:56 | Collect traces | RUNNING | Collection details: |
|        |              | & zip          |         |                     |
|        |              |                |         | Zip file:           |
|        |              |                |         | tfa_Tue_Mar_4_08_56 |
|        |              |                |         | _25_CET_2014.zip    |
|        |              |                |         | Tag:                |
|        |              |                |         | collection_Tue_Mar_ |
|        |              |                |         | 4_08_56_25_CET_2014 |
|        |              |                |         | _node_all           |
+--------+--------------+----------------+---------+---------------------+
| grac43 | Mar 04 08:56 | Run inventory  | RUNNING | RDBMS               |
|        |              |                |         | all:ASM:CRS:DBWLM:A |
|        |              |                |         | CFS:CRS:ASM:OS:INST |
|        |              |                |         | ALL:TNS:CHMOS       |
'--------+--------------+----------------+---------+---------------------'

Use tfactl to check for Errors and Startups

Check for errors:
# /u01/app/grid/tfa/bin/tfactl print errors 
++++++ Error Start +++++
Event Id          : GRACE232lppjmk62mrsu7si7ltbau98j
File Name         : /u01/app/oracle/diag/rdbms/grace2/GRACE21/trace/alert_GRACE21.log
Error Code        : ORA-1109
Error Description : ORA-1109 signalled during: ALTER DATABASE CLOSE NORMAL...
Error Time        : Mon Jul 22 12:37:06 CEST 2013
Startup Time          : Mon Jul 22 12:36:39 CEST 2013
Trace File Name   : NONE
++++++ Error End +++++
..
Check for database startups:
# /u01/app/grid/tfa/bin/tfactl print startups
++++++ Startup Start +++++
Event Id     : GRACE27rpp25v1c83sago9ohf1s6gn8s
File Name    : /u01/app/oracle/diag/rdbms/grace2/GRACE21/trace/alert_GRACE21.log
Startup Time : Mon Jul 22 12:32:34 CEST 2013
Dummy        : FALSE
++++++ Startup End +++++
++++++ Startup Start +++++
Event Id     : GRACE2nahm9evlmrtssl7etl36u7n25f
File Name    : /u01/app/oracle/diag/rdbms/grace2/GRACE21/trace/alert_GRACE21.log
Startup Time : Mon Jul 22 12:36:39 CEST 2013
Dummy        : FALSE
++++++ Startup End +++++

Collect tracefiles using TFA

# /u01/app/grid/tfa/bin/tfactl diagcollect -all -since 1h
Collecting data for all components using above parameters...
Running an inventory clusterwide ...
Run inventory completed locally ...
Collection name tfa_Thu_Sep_26_11_58_47_CEST_2013.zip
Sending diagcollect request to host : grac2
Sending diagcollect request to host : grac3
Getting list of files satisfying time range [Thu Sep 26 10:59:30 CEST 2013, Thu Sep 26 11:59:30 CEST 2013]
grac1: Zipping File: /u01/app/oracle/diag/rdbms/grace2/GRACE2_1/trace/GRACE2_1_dbrm_4375.trc
...
grac1: Zipping File: /u01/app/oracle/diag/rdbms/race2/RACE21/trace/alert_RACE21.log
Trimming file : /u01/app/oracle/diag/rdbms/race2/RACE21/trace/alert_RACE21.log with original file size : 255kB
Collecting extra files...
Total Number of Files checked : 2181
Total Size of all Files Checked : 1.7GB
Number of files containing required range : 41
Total Size of Files containing required range : 106MB
Number of files trimmed : 16
Total Size of data prior to zip : 43MB
Saved 77MB by trimming files
Zip file size : 1.9MB
Total time taken : 101s
Completed collection of zip files.

Logs are collected to:
/u01/app/grid/tfa/repository/collection_Thu_Sep_26_11_58_47_CEST_2013_node_all/grac1.tfa_Thu_Sep_26_11_58_47_CEST_2013.zip
/u01/app/grid/tfa/repository/collection_Thu_Sep_26_11_58_47_CEST_2013_node_all/grac2.tfa_Thu_Sep_26_11_58_47_CEST_2013.zip
/u01/app/grid/tfa/repository/collection_Thu_Sep_26_11_58_47_CEST_2013_node_all/grac3.tfa_Thu_Sep_26_11_58_47_CEST_2013.zip

Collect trace by a specific time
# /u01/app/grid/tfa/bin/tfactl diagcollect -all -from "Oct/18/2013 00:00:00" -to "Oct/18/2013 06:00:00" 

Collecting data for all components using above parameters...
Scanning files from Oct/18/2013 00:00:00 to Oct/18/2013 06:00:00
Running an inventory clusterwide ...

Collecting diagnostic data for a specific day
# $GRID_HOME/bin/tfactl diagcollect -all -for "Mar/22/2014"
Collecting data for all components using above parameters...
Collecting data for all nodes
Scanning files for Mar/22/2014
Repository Location in grac41 : /u01/app/grid/tfa/repository
2014/03/22 14:03:35 CET : Running an inventory clusterwide ...
2014/03/22 14:03:36 CET : Collection Name : tfa_Sat_Mar_22_14_03_29_CET_2014.zip
2014/03/22 14:03:43 CET : Sending diagcollect request to host : grac42
2014/03/22 14:03:43 CET : Sending diagcollect request to host : grac43 
....
Logs are collected to:
/u01/app/grid/tfa/repository/collection_Sat_Mar_22_14_03_29_CET_2014_node_all/grac41.tfa_Sat_Mar_22_14_03_29_CET_2014.zip
/u01/app/grid/tfa/repository/collection_Sat_Mar_22_14_03_29_CET_2014_node_all/grac42.tfa_Sat_Mar_22_14_03_29_CET_2014.zip
/u01/app/grid/tfa/repository/collection_Sat_Mar_22_14_03_29_CET_2014_node_all/grac43.tfa_Sat_Mar_22_14_03_29_CET_2014.zip

 

Add and remove directory

  • TFA directories may change due to switching from adming managed to policy managed
  • You run add database / add instance
  • Keep your TFA repository in sync with any changes
Remove directories:
# /u01/app/grid/tfa/bin/tfactl directory remove  /u01/app/oracle/diag/rdbms/grace2/GRACE2_1/trace
# /u01/app/grid/tfa/bin/tfactl directory remove /u01/app/oracle/diag/rdbms/grace2/GRACE21/trace      

Add a directory:
# /u01/app/grid/tfa/bin/tfactl directory  add  /u01/app/oracle/diag/rdbms/grace2/grac21/trace

 

Add OSWatcher archive for TFA processing

Check TFA archive location:
$  ps -ef | grep OSWatcher  | grep -v grep
root     11018     1  0 14:51 ?        00:00:00 /bin/sh ./OSWatcher.sh 10 60 gzip /home/OSW/oswbb/archive
root     12028 11018  0 14:51 ?        00:00:00 /bin/sh ./OSWatcherFM.sh 60 /home/OSW/oswbb/archive
-->  /home/OSW/oswbb/archive is OSWatcher tracefile location

Run now on each instance:
# /u01/app/grid/tfa/bin/tfactl directory add /home/OSW/oswbb/archive

Add CHM directory

  • Note you should not need to add CHM directory as this directory should be already availabe
  • Check with  $GRID_HOME/tfa/bin/tfactl  print directories  | grep crf
First check whether a crf directory is already available
$  $GRID_HOME/tfa/bin/tfactl  print directories  | grep crf
| /u01/app/11204/grid/log/grac41/crf | CRS                                    | public     | root     |
If your get above output - stop here as the directory is already listed

Find CHM repository directory
Note : ologgerd is only running on a single node - Read following link to find this node. 

$ ps -ef | grep ologgerd | grep -v grep
root     16996     1  0 01:06 ?        00:04:19 /u01/app/11203/grid/bin/ologgerd -m grac2 -r -d /u01/app/11203/grid/crf/db/grac1

Add directory to TFA
# /u01/app/grid/tfa//bin/tfactl directory add /u01/app/11203/grid/crf/db/grac1
Failed to add directory to TFA. Unable to determine parameters for directory: /u01/app/11203/grid/crf/db/grac1

Please enter component for this Directory [RDBMS|CRS|ASM|INSTALL|OS|CFGTOOLS] : OS
Running Inventory ...
.----------------------------------------------------------------------------------------------------------------------------.
| Trace Directory                                                   | Component                      | Permission | Added By |
+-------------------------------------------------------------------+--------------------------------+------------+----------+
| /etc/oracle                                                       | CRS                            | public     | root     |
+-------------------------------------------------------------------+--------------------------------+------------+----------+
| /home/OSW/oswbb/archive                                           | OS                             | public     | root     |
+-------------------------------------------------------------------+--------------------------------+------------+----------+
| /u01/app/11203/grid/cfgtoollogs/opatch                            | INSTALL                        | public     | root     |
+-------------------------------------------------------------------+--------------------------------+------------+----------+
| /u01/app/11203/grid/crf/db/grac1                                  | OS                             | public     | root     |
+-------------------------------------------------------------------+--------------------------------+------------+----------+

 

Verify TFA repository space

# /u01/app/grid/tfa/bin/tfactl print repository
.-----------------------------------------------------.
|                        grac1                        |
+----------------------+------------------------------+
| Repository Parameter | Value                        |
+----------------------+------------------------------+
| Location             | /u01/app/grid/tfa/repository |
| Maximum Size (MB)    | 2092                         |
| Current Size (MB)    | 156                          |
| Status               | OPEN                         |
'----------------------+------------------------------'

.-----------------------------------------------------.
|                        grac3                        |
+----------------------+------------------------------+
| Repository Parameter | Value                        |
+----------------------+------------------------------+
| Location             | /u01/app/grid/tfa/repository |
| Maximum Size (MB)    | 2273                         |
| Current Size (MB)    | 67                           |
| Status               | OPEN                         |
'----------------------+------------------------------'

.-----------------------------------------------------.
|                        grac2                        |
+----------------------+------------------------------+
| Repository Parameter | Value                        |
+----------------------+------------------------------+
| Location             | /u01/app/grid/tfa/repository |
| Maximum Size (MB)    | 1981                         |
| Current Size (MB)    | 60                           |
| Status               | OPEN                         |
'----------------------+------------------------------'

Setting TFA tracelevel and review trace file

# /u01/app/grid/tfa/bin/tfactl set tracelevel=4 -c
Running on Host : grac1 
Running Check through Java CLI
Opening Port file /u01/app/grid/tfa/grac1/tfa_home/internal/port.txtOpening Parameter file /u01/app/grid/tfa/grac1/tfa_home/tfa_setup.txtWe got : CheckOK
...
'-----------------------------------------+---------'
| Configuration Parameter                 | Value   |
+-----------------------------------------+---------+
| TFA version                             | 2.5.1.5 |
| Automatic diagnostic collection         | OFF     |
| Trimming of files during diagcollection | ON      |
| Repository current size (MB) in grac1   | 131     |
| Repository maximum size (MB) in grac1   | 2629    |
| Trace level                             | 4       |
'-----------------------------------------+---------'
#### Done ####

Tracefile name and location
# pwd
/u01/app/grid/tfa/grac3/tfa_home/log
#  ls -rlt
-rw-r--r-- 1 root root     3641 Oct 23 10:26 syserrorout
-rw-r--r-- 1 root root   134679 Oct 23 10:28 diagcollect.log
-rw-r--r-- 1 root root 25108979 Oct 23 10:32 tfa.10.22.2013-19.54.37.log

With level 4 tracing we see following message for a tracefile added ZIP file
10.23.2013-10.26.30 -- MESSAGE : ZipTracesForDatesThread Transferring from /u01/app/oracle/diag/rdbms/grace2/grac23/trace/alert_grac23.log to /u01/app/grid/tfa/repository/temp_1382516788278/collections/alert_grac23.log

Uninstall TFA

# cd  /u01/app/grid/tfa/grac1/tfa_home/bin
# ./uninstalltfa.sh
Stopping TFA in grac1...
Shutting down TFA
oracle-tfa stop/waiting
Killing TFA running with pid 15860
Successfully shutdown TFA..
Stopping TFA in grac2 and removing /u01/app/grid/tfa/grac2/tfa_home...
Removing TFA from grac2...
Stopping TFA in grac2...
Shutting down TFA
oracle-tfa stop/waiting
Killing TFA running with pid 3251
Successfully shutdown TFA..
Deleting TFA support files on grac2:
Removing /etc/init.d/init.tfa...
Removing /u01/app/grid/tfa/bin...
Removing /u01/app/grid/tfa/grac2...
Stopping TFA in grac3 and removing /u01/app/grid/tfa/grac3/tfa_home...
Removing TFA from grac3...
Stopping TFA in grac3...
Shutting down TFA
oracle-tfa stop/waiting
Killing TFA running with pid 1615
Successfully shutdown TFA..
Deleting TFA support files on grac3:
Removing /etc/init.d/init.tfa...
Removing /u01/app/grid/tfa/bin...
Removing /u01/app/grid/tfa/grac3...
Deleting TFA support files on grac1:
Removing /etc/init.d/init.tfa...
Removing /u01/app/grid/tfa/bin...
Removing /u01/app/grid/tfa/grac1...

Upgrade TFA 2.5 to 3.1

  • If you install SupportBundle_v1_3 you will get TFA 3.1
  • 11.2.0.4 has installed TFA 2.5.1.5  per default
  •  Reinstall TFA 11.2.0.4 in a 11.2.0.4 GRID_HOME  after a downgrade
     /u01/app/grid/11.2.0.4/crs/install/tfa_setup.sh -silent -crshome /u01/app/grid/11.2.0.4
Extract zip file from SupportBundle_v1_3 and run 
# ./installTFALite
Starting TFA installation
TFA is already installed. Patching /u01/app/11204/grid/tfa/grac41/tfa_home...
TFA will be Patched on: 
grac41
grac42
grac43
Do you want to continue with patching TFA? [Y|N] [Y]: 
Checking for ssh equivalency in grac42
grac42 is configured for ssh user equivalency for root user
Checking for ssh equivalency in grac43
grac43 is configured for ssh user equivalency for root user
Auto patching is enabled in grac41
Key Stores are already updated in grac42
Key Stores are already updated in grac43
Shutting down TFA for Patching...
Shutting down TFA
oracle-tfa stop/waiting
. . . . . 
Killing TFA running with pid 18946
. . . 
Successfully shutdown TFA..
Renaming /u01/app/11204/grid/tfa/grac41/tfa_home/jar to /u01/app/11204/grid/tfa/grac41/tfa_home/jlib
Adding INSTALL_TYPE = GI to tfa_setup.txt
Copying /u01/app/11204/grid/tfa/grac41/tfa_home/output/ to /u01/app/grid/tfa/grac41/
The current version of Berkeley DB is 4.0.103
Copying je-4.1.27.jar to /u01/app/11204/grid/tfa/grac41/tfa_home/jlib/
Copying je-5.0.84.jar to /u01/app/11204/grid/tfa/grac41/tfa_home/jlib/
Running DbPreUpgrade_4_1 utility
Output of upgrade : Pre-upgrade succeeded
Creating ZIP: /u01/app/11204/grid/tfa/grac41/tfa_home/internal/tfapatch.zip
Running commands to fix init.tfa and tfactl in localhost
Starting TFA in grac41...
Starting TFA..
oracle-tfa start/running, process 9222
Waiting up to 100 seconds for TFA to be started..
. . . . . 
Successfully started TFA Process..
. . . . . 
TFA Started and listening for commands
Removing /u01/app/11204/grid/tfa/grac41/tfa_home/jlib/je-4.0.103.jar
Enabling Access for Non-root Users on grac41...
Adding default users and groups to TFA Access list...
Using SSH to patch TFA to remote nodes:

Applying Patch on grac42:
...

Applying Patch on grac43:
..

When running tfactl diagcollect like you will get following Inventory Status message
# $GRID_HOME/tfa/bin/tfactl diagcollect -all -since 1h
# $GRID_HOME/tfa/bin/tfactl  print status
.-----------------------------------------------------------------------------------------.
| Host   | Status of TFA | PID   | Port | Version | Build ID           | Inventory Status |
+--------+---------------+-------+------+---------+--------------------+------------------+
| grac42 | RUNNING       |  3951 | 5000 |     3.1 | 310020140205043544 | RUNNING          |
| grac41 | RUNNING       |  4201 | 5000 |     3.1 | 310020140205043544 | RUNNING          |
| grac43 | RUNNING       | 22284 | 5000 |     3.1 | 310020140205043544 | RUNNING          |
'--------+---------------+-------+------+---------+--------------------+------------------'
  • Note it will take some time until Inventory status goes to completed
  • You can monitory outstanding actions by running:   #  $GRID_HOME/tfa/bin/tfactl  print actions

Check TFA for integration of CHM and OSWatcher Output

Script check_tools.sh
#!/bin/sh
#
# OSWatcher location: /u01/app/11204/grid/oswbb -> search string oswbb
#     CHM   location: /u01/app/11204/grid/crf   -> search string crf
#
OSWatcher_loc_string="oswbb"
CHM_loc_string="crf"

echo  "-> OSWatcher status grac41:"
sudo ssh grac41 "ps -elf | grep OSWatcher  | grep -v grep"
echo  "-> OSWatcher status grac42:"
ssh grac42 "ps -elf | grep OSWatcher  | grep -v grep"
echo  "-> OSWatcher status grac43:"
ssh grac43 "ps -elf | grep OSWatcher  | grep -v grep"

echo "-> CHM Mater/Replica info"
ssh grac41 "$GRID_HOME/bin/oclumon manage -get MASTER REPLICA"
echo "-> CHM state "
ssh grac41 "$GRID_HOME/bin/crsctl status res ora.crf -init | grep STATE"
ssh grac42 "$GRID_HOME/bin/crsctl status res ora.crf -init | grep STATE"
ssh grac43 "$GRID_HOME/bin/crsctl status res ora.crf -init | grep STATE"

echo "-> TFA print status and open actions  "
ssh grac41 $GRID_HOME/tfa/bin/tfactl print status
ssh grac41 $GRID_HOME/tfa/bin/tfactl print actions  
echo "-> TFA directories for OSWatcher/CHM  grac41"
echo "-> TFA directory search string: $OSWatcher_loc_string|$CHM_loc_string"
ssh grac41 $GRID_HOME/tfa/bin/tfactl  print directories  | egrep "$OSWatcher_loc_string|$CHM_loc_string"
echo "-> TFA directories for OSWatcher/CHM  grac42"
ssh grac42 $GRID_HOME/tfa/bin/tfactl  print directories  | egrep "$OSWatcher_loc_string|$CHM_loc_string"
echo "-> TFA directories for OSWatcher/CHM  grac43"
ssh grac43 $GRID_HOME/tfa/bin/tfactl  print directories  | egrep "$OSWatcher_loc_string|$CHM_loc_string

Output from check_tools.sh script running on a 11.2.0.4 RAC system
[grid@grac41 ~]$ ./check_tools.sh
-> OSWatcher status grac41:
[sudo] password for grid: 
0 S root     15448     1  0  80   0 - 26670 wait   13:36 ?        00:00:00 /bin/sh ./OSWatcher.sh 48 10 gzip /u01/app/11204/grid/oswbb/archive
0 S root     16274 15448  0  80   0 - 26535 wait   13:36 ?        00:00:00 /bin/sh ./OSWatcherFM.sh 10 /u01/app/11204/grid/oswbb/archive
-> OSWatcher status grac42:
0 S root     15230     1  0  80   0 - 26669 wait   13:38 ?        00:00:00 /bin/sh ./OSWatcher.sh 48 10 gzip /u01/app/11204/grid/oswbb/archive
0 S root     15970 15230  0  80   0 - 26535 wait   13:38 ?        00:00:00 /bin/sh ./OSWatcherFM.sh 10 /u01/app/11204/grid/oswbb/archive
-> OSWatcher status grac43:
0 S root     21210     1  0  80   0 - 26668 wait   13:23 ?        00:00:01 /bin/sh ./OSWatcher.sh 48 10 gzip /u01/app/11204/grid/oswbb/archive
0 S root     21860 21210  0  80   0 - 26535 wait   13:24 ?        00:00:00 /bin/sh ./OSWatcherFM.sh 10 /u01/app/11204/grid/oswbb/archive
-> CHM Mater/Replica info
Master = grac43
Replica = grac42
 Done 
-> CHM state 
STATE=ONLINE on grac41
STATE=ONLINE on grac42
STATE=ONLINE on grac43
-> TFA print status and open actions  

.-----------------------------------------------------------------------------------------.
| Host   | Status of TFA | PID   | Port | Version | Build ID           | Inventory Status |
+--------+---------------+-------+------+---------+--------------------+------------------+
| grac41 | RUNNING       |  4201 | 5000 |     3.1 | 310020140205043544 | COMPLETE         |
| grac42 | RUNNING       |  3951 | 5000 |     3.1 | 310020140205043544 | COMPLETE         |
| grac43 | RUNNING       | 22284 | 5000 |     3.1 | 310020140205043544 | COMPLETE         |
'--------+---------------+-------+------+---------+--------------------+------------------'
.------------------------------------------.
| HOST | TIME | ACTION | STATUS | COMMENTS |
+------+------+--------+--------+----------+
'------+------+--------+--------+----------'

-> TFA directories for OSWatcher/CHM  grac41
-> TFA directory search string: oswbb|crf
| /u01/app/11204/grid/log/grac41/crf | CRS                                    | public     | root     |
| /u01/app/11204/grid/log/grac41/crf | CRS                                    | public     | root     |
| /u01/app/11204/grid/oswbb/archive  | OS                                     | private    | root     |
-> TFA directories for OSWatcher/CHM  grac42
| /u01/app/11204/grid/log/grac42/crf | CRS                  | public     | root     |
| /u01/app/11204/grid/log/grac42/crf | CRS                  | public     | root     |
| /u01/app/11204/grid/oswbb/archive  | OS                   | private    | root     |
-> TFA directories for OSWatcher/CHM  grac43
| /u01/app/11204/grid/log/grac43/crf | CRS                  | public     | root     |
| /u01/app/11204/grid/log/grac43/crf | CRS                  | public     | root     |
| /u01/app/11204/grid/oswbb/archive  | OS                   | private    | root     |

OSWatcher – Installation and Usage

Install OSWatcher

  • Download OSWatcher from OTN and Untar oswatcher ( see Note Doc ID 301137.1 )
  • Untar the reladet TAR archive : tar xvf oswbb601.tar
  • Before starting OSWatcher is checking for a process with OSWatcher string in its full path .
  • Don’t install OSWatcher in a directory named OSWatcher and use full PATH to start the tool. Even running a gedit session (  $ gedit OSWatcher.dat ) will signal that OSWatcher is running and you can’t start OSWatcher
  • In short :  ps -e | grep OSWatch  should not return any results before starting OSWatcher
# tar xvf oswbb601.tar 
oswbb/
oswbb/src/
oswbb/src/tombody.gif
oswbb/src/Thumbs.db
oswbb/src/missing_graphic.gif

 

Create file private.net for monitoring Cluster interconnet



Create file private.net based on Exampleprivate.net - here is the Linux Version 

#####################################################################
# This file contains examples of how to monitor private networks. To
# monitor your private networks create an executable file in this same
# directory named private.net. Use the example for your host os below.
# Make sure not to remove the last line in this file. Your file
# private.net MUST contain the rm lock.file line.
######################################################################
#Linux Example
######################################################################
echo "zzz ***"`date`
traceroute -r -F grac1int.example.com
traceroute -r -F grac2int.example.com
traceroute -r -F grac3int.example.com
######################################################################
rm locks/lock.file