How to Delete From or Add Resource to OCR in Oracle Clusterware

This document explains necessary steps to remove/add resource from/to OCR when normal srvctl command fails. 

Common scenario is resource in "UNKNOWN" status and can not be stopped or deleted.

A. Terms/Variables used in this note

RESOURCE_TYPE can be database, instance, service, listener etc

RESOURCE_NAME is the name from crs_stat command for pre-11.2, or the name from "crsctl stat res" for 11gR2, i.e. ora.racdb.db

RESOURCE_HOME refers to the ORACLE_HOME that the resource runs off, i.e. RDBMS_HOME for database, instance and service resource; 

RESOURCE_OWNER refers to the OS user that owns the RESOURCE_HOME, i.e. grid user for GRID_HOME.

B. Resource must be managed by srvctl from RESOURCE_HOME/bin

Resource must be managed by srvctl from $RESOURCE_HOME where resource is running off, for example, vip from $GRID_HOME or $CRS_HOME, 11.2 .db in 11.2 RDBMS_HOME, and 11.1 .db in 11.1 RDBMS_HOME etc.

If wrong srvctl is used, the following error will be reported:

PRCD-1245 : Addition of database version 11.2.0.3.0 is not allowed using srvctl version 10.2.0.0.0

OR

PRCD-1027 : Failed to retrieve database racdb
PRKP-1088 : Failed to retrieve configuration of cluster database racdb
PRKR-1078 : Database racdb of version 10.2.0.0.0 cannot be administered using current version of srvctl. Instead run srvctl from /dbhome/10.2

OR

PRCD-1027 : Failed to retrieve database racdb
PRCD-1229 : An attempt to access configuration of database racdb was rejected because its version 11.2.0.2.0 differs from the program version 11.2.0.3.0. Instead run the program from /dbhome/11.2.0.2

C. To remove a resource from OCR

A resource should be in OFFLINE state before it can be removed, and srvctl from RESOURCE_HOME should be executed as RESOURCE_OWNER:

1. To stop:

Try the following sequentially until the resource is stopped successfully:

$ $RESOURCE_HOME/bin/srvctl stop <RESOURCE_TYPE> <options>
$ $RESOURCE_HOME/bin/srvctl stop <RESOURCE_TYPE> <options> -f

For srvctl syntax, refer to:

Oracle Real Application Clusters
Administration and Deployment Guide

Server Control Utility Reference


2. To remove 

Once the resource is stopped, try the following sequentially until resource is removed successfully:

$ $RESOURCE_HOME/bin/srvctl remove<RESOURCE_TYPE><options>
$ $RESOURCE_HOME/bin/srvctl remove<RESOURCE_TYPE><options> -f

D. To add a resource to OCR

3. To add:

$ $RESOURCE_HOME/bin/srvctl add<RESOURCE_TYPE><options>

E. To troubleshoot

If srvctl reports error, the following SRVM tracing can be turned on before executing the srvctl command:

$ script /tmp/out.1
$ SRVM_TRACE=true
$ export SRVM_TRACE
$ $RESOURCE_HOME/bin/srvctl <command> <RESOURCE_TYPE> <option>
$ exit

Screen output will be saved in /tmp/out.1

To get help on srvctl syntax:

$ $RESOURCE_HOME/bin/srvctl <command> <RESOURCE_TYPE> -h

E.1. Engage Oracle Support

If the issue can not be solved, engage Oracle Support

F. Misc

F1. VIP or network resource(in 11.2.0.2 or above)

Addition, removal or modification of VIP or network resource (in 11.2.0.2 or above, ora.net1.network) must be done by root user

F2. Pre-11.2 database in Oracle Restart

Pre-11.2 single instance databases can not be managed by 11gR2 Grid Infrastructure Standalone (aka Oracle Restart), the following error will be reported when trying to register:

PRCD-1245 : Addition of database version n.n.0.n.0 is not allowed using srvctl version n.n.n.n.n

OR

srvctl[nnnn]: /bin/java:  not found"

F3. listener can only be managed by netca in 10gR2

F4. DB_UNIQUE_NAME

When managing database with "srvctl <command> database -d <dbname> <options>", if dbname is different than db_unique_name, db_unique_name must be used to avoid error PRCD-1120 PRCR-1001

REFERENCES

NOTE:1050908.1 - Troubleshoot Grid Infrastructure Startup Issues
NOTE:1068835.1 - What to Do if 11gR2 Grid Infrastructure is Unhealthy

NOTE:948456.1 - Pre 11.2 Database Issues in 11gR2 Grid Infrastructure Environment