Jump to content

RAC Attack - Oracle Cluster Database at Home/RAC Attack 12c/Clusterware Callouts

Add links
From Wikibooks, open books for an open world


The goal of this lab is to demonstrate Oracle Fast Application Notification (FAN) Callouts. In versions prior to 11g, these were also known as Oracle Clusterware Callouts.

This feature is a relatively little-known capability for Oracle Clusterware to fire a script (or a whole directory full of them) to perform whatever tasks you may want performed when a cluster-wide event happens.

For more information, consult the documentation here: http://docs.oracle.com/cd/E16655_01/rac.121/e17887/hafeats.htm#BABHAJAH

For this exercise, we’ll configure some FAN callout scripts on each node and then trigger various cluster events to see how each one triggers the callout script.


  1. Start with a normal, running cluster with both nodes up and running.
  2. From a shell prompt (logged in as oracle) on each server, navigate to /u01/app/12.1.0/grid/racg/usrco. Create file there called callout1.sh using vi (or your favorite editor). The contents of the file should be this:
    3. #!/bin/ksh umask 022 FAN_LOGFILE=/tmp/`hostname`_uptime.log echo $* "reported="`date` >> $FAN_LOGFILE &
  3. Make sure that the permissions on the file are set to 755 using the following command:
    4. [oracle@<node_name> ~]$ chmod 755 /u01/app/12.1.0/grid/racg/usrco/callout1.sh
  4. Monitor the logfiles for clusterware on each node. On each node, start a new window and run the following command:
    5. [oracle@<node_name> ~]$ tail -f /u01/app/12.1.0/grid/log/`hostname -s`/crsd/crsd.log
  5. Next, we need to trigger an event that will cause the callout to fire. One such event is node shutdown. Shutdown the clusterware on node collabn2.
    6. [root@collabn2 ~]# crsctl stop crs CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'collabn2' CRS-2673: Attempting to stop 'ora.crsd' on 'collabn2' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'colleabn2' CRS-2673: Attempting to stop 'ora.rac.db' on 'collabn2' CRS-2673: Attempting to stop 'ora.DATA.SHARED.advm' on 'collabn2' CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'collabn2' ... CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'collabn2' has completed CRS-4133: Oracle High Availability Services has been stopped.
  6. Following this command, watch the logfiles you began monitoring in step 2 above. Because we set long timeouts on our test cluster, you might have to wait for a few minutes before you see anything.
    • You should eventually observe entries noting that the node has failed and shortly following that, you should observe an entry placed in the /tmp/<hostname>_uptime.log file indicating that the node is down.
    • Note which members run the clusterware callout script. (A surviving member could run commands to notify clients and/or application servers that one of the cluster nodes has died.)
    You should see these messages in the /tmp/*.log files: NODE VERSION=2.0 host=collabn2 status=nodedown reason=public_nw_down incarn=0 timestamp=2013-08-13 12:15:51 timezone=-04:00 vip_ips= reported=Tue Aug 13 12:15:51 EDT 2013 INSTANCE VERSION=1.0 service=rac.racattack database=rac instance=RAC2 host=collabn2 status=down reason=USER timestamp=2013-08-13 12:15:51 timezone=-04:00 db_domain=racattack reported=Tue Aug 13 12:15:51 EDT 2013 NODE VERSION=2.0 host=collabn2 status=nodedown reason=public_nw_down incarn=0 timestamp=2013-08-13 12:15:51 timezone=-04:00 vip_ips= reported=Tue Aug 13 12:15:52 EDT 2013 NODE VERSION=1.0 host=collabn2 status=nodedown reason=member_leave incarn=271826344 timestamp=2013-08-13 12:17:23 timezone=-04:00 reported=Tue Aug 13 12:17:23 EDT 2013 NODE VERSION=1.0 host=collabn2 incarn=271826344 status=down reason=member_leave timestamp=13-Aug-2013 12:17:23 reported=Tue Aug 13 12:17:24 EDT 2013
  7. Restart the clusterware. Is there a node up event?
    8. [root@collabn2 bin]# crsctl start crs
  8. Try powering off one of the virtual machines – is there an difference from the previous test? What if you disable a linux network interface or virtual network card?
  9. You may conduct more testing, if you wish. Another interesting event is a database instance going down unexpectedly. Come back to this lab after installing a database to test that situation.
    10. [oracle@collabn2 ~]$ sqlplus "/ as sysdba" SQL*Plus: Release 12.1.0.1.0 Production on Tue Aug 13 12:34:52 2013 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to: Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Advanced Analytics and Real Application Testing options SQL> shutdown abort; ORACLE instance shut down. SQL>


Retrieved from "https://en.wikibooks.org/wiki/RAC_Attack_-_Oracle_Cluster_Database_at_Home/RAC_Attack_12c/Clusterware_Callouts"