Oracle 12.1 – Find OCR Master Node

I’ve been dealing with a problem where, sometimes, a rebooted RAC node is unable to join back the cluster. The issue seems to be with the “Master Node”, which refuses to accept the node.

So I’ve to know which is the “Master Node” (the current known solution is to reboot it, and then all nodes join the cluster).

There is the Oracle note: How to Find OCR Master Node (Doc ID 1281982.1)

And there is this blog entry: 11G R2 RAC: How to identify the master node in RAC

In my case I’m using Oracle, the location of the files is a bit different. The location of the OCR Master Node can be found on this version using one of the following ways:

  • Check the crsd logs for “OCR MASTER”
grep "OCR MASTER" ${ORACLE_BASE}/diag/crs/`hostname`/crs/trace/crsd*

and, if the logs did not rotate too much yet, you should see one of the two below:

/u00/app/oracle/diag/crs/anjovm1/crs/trace/crsd_73.trc:2018-01-13 14:05:30.535186 :  OCRMAS:3085: th_master:13: I AM THE NEW OCR MASTER at incar 2. Node Number 1

/u00/app/oracle/diag/crs/anjovm2/crs/trace/crsd_71.trc:2018-01-13 14:05:32.823231 :  OCRMAS:3085: th_master: NEW OCR MASTER IS 1
  • Check the location of the OCR automatic backups

the cluster node currently keeping the backups, is the OCR master node. If you see older backups on other nodes, it was when they were OCR master nodes on its turn.

ls -l /u00/app/<cluster_name>
-rw-r--r--    1 root     system      943266 Jan  14 00:01 backup00.ocr
-rw-r--r--    1 root     system      943266 Jan 13 20:01 backup01.ocr
-rw-r--r--    1 root     system      943266 Jan 13 16:01 backup02.ocr
-rw-r--r--    1 root     system      943266 Jan 13 00:00 day.ocr
-rw-r--r--    1 root     system      943266 Jan  14 00:01 day_.ocr
-rw-r--r--    1 root     system      943266  Dec 31 23:55 week.ocr
-rw-r--r--    1 root     system      943266 Jan 07 23:59 week_.ocr

Note: do not confuse the OCR master node with the Cluster Health Monitor repository master node, which you get using the command:

/u00/app/ manage -get MASTER


Add path to ASMCMD prompt

Just discovered this morning how to show the path on the ASMCMD tool and could not be easier:

anjovm2:/home/grid:+ASM2 $ asmcmd -p
ASMCMD [+] > cd dgdata
ASMCMD [+dgdata] > cd db12c
ASMCMD [+dgdata/db12c] > ls
ASMCMD [+dgdata/db12c] > cd datafile
ASMCMD [+dgdata/db12c/datafile] >

crs_stat and Oracle cluster resources status

One script output that Oracle to my knowledge did not improve yet was the “crs_stat -t”. On 11.2 there is the “./crsctl status resource -t“, but the output is quite long.

The usual output is like this:

hqbuun415:/oracle/app/product/11.1/crs/bin:crs:$crs_stat -t
Name           Type           Target    State     Host
ora....ISP1.cs application    ONLINE    ONLINE    hqbuun415
ora....ss2.srv application    ONLINE    ONLINE    hqbuun415
ora....ISP2.cs application    ONLINE    ONLINE    hqbuun414
ora....ss1.srv application    ONLINE    ONLINE    hqbuun414
ora....OAFO.cs application    ONLINE    ONLINE    hqbuun414
ora....ss1.srv application    ONLINE    ONLINE    hqbuun414
ora....M3PL.cs application    ONLINE    ONLINE    hqbuun415
ora....ss2.srv application    ONLINE    ONLINE    hqbuun415
ora....MALF.cs application    ONLINE    ONLINE    hqbuun414
ora....ss1.srv application    ONLINE    ONLINE    hqbuun414
ora....MAVE.cs application    ONLINE    ONLINE    hqbuun415
ora....ss2.srv application    ONLINE    ONLINE    hqbuun415
ora....WMEA.cs application    ONLINE    ONLINE    hqbuun414
ora....ss1.srv application    ONLINE    ONLINE    hqbuun414
ora....WMSS.db application    ONLINE    ONLINE    hqbuun414
ora....s1.inst application    ONLINE    ONLINE    hqbuun414
ora....s2.inst application    ONLINE    ONLINE    hqbuun415
ora....SM1.asm application    ONLINE    ONLINE    hqbuun414
ora....14.lsnr application    ONLINE    ONLINE    hqbuun414
ora....414.gsd application    ONLINE    ONLINE    hqbuun414
ora....414.ons application    ONLINE    ONLINE    hqbuun414
ora....414.vip application    ONLINE    ONLINE    hqbuun414
ora....SM2.asm application    ONLINE    ONLINE    hqbuun415
ora....15.lsnr application    ONLINE    ONLINE    hqbuun415
ora....415.gsd application    ONLINE    ONLINE    hqbuun415
ora....415.ons application    ONLINE    ONLINE    hqbuun415
ora....415.vip application    ONLINE    ONLINE    hqbuun415

But on Metalink (My Oracle Support) note 259301.1 we can find the magic script (which works still with Oracle 11.2):

--------------------------- Begin Shell Script -------------------------------

# Sample 10g CRS resource status query script
# Description:
#    - Returns formatted version of crs_stat -t, in tabular
#      format, with the complete rsc names and filtering keywords
#   - The argument, $RSC_KEY, is optional and if passed to the script, will
#     limit the output to HA resources whose names match $RSC_KEY.
# Requirements:
#   - $ORA_CRS_HOME should be set in your environment 

AWK=/usr/xpg4/bin/awk    # if not available use /usr/bin/awk

# Table header:echo ""
  'BEGIN {printf "%-45s %-10s %-18sn", "HA Resource", "Target", "State";
          printf "%-45s %-10s %-18sn", "-----------", "------", "-----";}'

# Table body:
$ORA_CRS_HOME/bin/crs_stat $QSTAT | $AWK 
 'BEGIN { FS="="; state = 0; }
  $1~/NAME/ && $2~/'$RSC_KEY'/ {appname = $2; state=1};
  state == 0 {next;}
  $1~/TARGET/ && state == 1 {apptarget = $2; state=2;}
  $1~/STATE/ && state == 2 {appstate = $2; state=3;}
  state == 3 {printf "%-45s %-10s %-18sn", appname, apptarget, appstate; state=0;}'
--------------------------- End Shell Script -------------------------------

This script makes a clear output as:

HA Resource                                   Target     State
-----------                                   ------     -----
ora.WMS01_WMSS.EISP1.cs                       ONLINE     ONLINE on hqbuun415
ora.WMS01_WMSS.EISP1.wmss2.srv                ONLINE     ONLINE on hqbuun415
ora.WMS01_WMSS.EISP2.cs                       ONLINE     ONLINE on hqbuun414
ora.WMS01_WMSS.EISP2.wmss1.srv                ONLINE     ONLINE on hqbuun414
ora.WMS01_WMSS.NESOAFO.cs                     ONLINE     ONLINE on hqbuun414
ora.WMS01_WMSS.NESOAFO.wmss1.srv              ONLINE     ONLINE on hqbuun414
ora.WMS01_WMSS.WM3PL.cs                       ONLINE     ONLINE on hqbuun415
ora.WMS01_WMSS.WM3PL.wmss2.srv                ONLINE     ONLINE on hqbuun415
ora.WMS01_WMSS.WMALF.cs                       ONLINE     ONLINE on hqbuun414
ora.WMS01_WMSS.WMALF.wmss1.srv                ONLINE     ONLINE on hqbuun414
ora.WMS01_WMSS.WMAVE.cs                       ONLINE     ONLINE on hqbuun415
ora.WMS01_WMSS.WMAVE.wmss2.srv                ONLINE     ONLINE on hqbuun415
ora.WMS01_WMSS.WMEA.cs                        ONLINE     ONLINE on hqbuun414
ora.WMS01_WMSS.WMEA.wmss1.srv                 ONLINE     ONLINE on hqbuun414
ora.WMS01_WMSS.db                             ONLINE     ONLINE on hqbuun414
ora.WMS01_WMSS.wmss1.inst                     ONLINE     ONLINE on hqbuun414
ora.WMS01_WMSS.wmss2.inst                     ONLINE     ONLINE on hqbuun415
ora.hqbuun414.ASM1.asm                        ONLINE     ONLINE on hqbuun414
ora.hqbuun414.LISTENER_HQBUUN414.lsnr         ONLINE     ONLINE on hqbuun414
ora.hqbuun414.gsd                             ONLINE     ONLINE on hqbuun414
ora.hqbuun414.ons                             ONLINE     ONLINE on hqbuun414
ora.hqbuun414.vip                             ONLINE     ONLINE on hqbuun414
ora.hqbuun415.ASM2.asm                        ONLINE     ONLINE on hqbuun415
ora.hqbuun415.LISTENER_HQBUUN415.lsnr         ONLINE     ONLINE on hqbuun415
ora.hqbuun415.gsd                             ONLINE     ONLINE on hqbuun415
ora.hqbuun415.ons                             ONLINE     ONLINE on hqbuun415
ora.hqbuun415.vip                             ONLINE     ONLINE on hqbuun415