The Generic Server Pool

The Generic Server Pool

The Generic server pool stores pre-11g release 2 (11.2) Oracle Databases and administrator-managed databases that have fixed configurations. Additionally, the Generic server pool contains servers that match either of the following:

• Servers that you specified in the HOSTING_MEMBERS resource attribute of all resources of the application resource type

·         Servers with names you specified in the SERVER_NAMES attribute of the server pools that list the Generic server pool as a parent server pool

The Generic server pool's attributes are restricted, as follows:

·         No one can modify configuration attributes of the Generic server pool (all attributes are read-only)

·         When you specify a server name in the HOSTING_MEMBERS resource attribute, Oracle Clusterware only allows it if the server is:

o    Online and exists in the Generic server pool

o    Online and exists in the Free server pool, in which case Oracle Clusterware moves the server into the Generic server pool

o    Online and exists in any other server pool and the client is either a cluster administrator or is allowed to use the server pool's servers, in which case, the server is moved into the Generic server pool

o    Offline and the client is a cluster administrator

·         When you register a child server pool with the Generic server pool, Oracle Clusterware only allows it if the server names pass the same requirements as previously specified for the resources.

Servers are initially considered for assignment into the Generic server pool at cluster startup time or when a server is added to the cluster, and only after that to other server pools.

Creating an Application VIP Managed by Oracle Clusterware

Creating an Application VIP Managed by Oracle Clusterware

If clients of an application access the application through a network, and the placement policy for the application allows it to fail over to another node, then you must register a virtual internet protocol address (VIP) on which the application depends. An application VIP is a cluster resource that Oracle Clusterware manages (Oracle Clusterware provides a standard VIP agent for application VIPs). 

 

You should base any new application VIPs on this VIP type to ensure that your system experiences consistent behavior among all of the VIPs that you deploy in your cluster.

While you can add a VIP in the same way that you can add any other resource that Oracle Clusterware manages, Oracle recommends using the script Grid_home/bin/appvipcfg to create or delete an application VIP.

To create an application VIP, use the following syntax:


appvipcfg create -network=network_number -ip=ip_address -vipname=vip_name -user=user_name [-group=group_name] [-failback=0 | 1]


To delete an application VIP, use the following syntax:


appvipcfg delete -vipname=vip_name

Where network_number is the number of the network, ip_address is the IP address, vip_name is the name of the VIP, user_name is the name of the user who installed Oracle Database, and group_name is the name of the group. The default value of the -failback option is 0.

 

If you set the option to 1, then the VIP (and therefore any resources that depend on VIP) fails back to the original node when it becomes available again.

For example, as root, run the following command:


# Grid_home/bin/appvipcfg create -network=1 -ip=148.87.58.196 -vipname=appsVIP -user=root

The script only requires a network number (default is 1), the IP address, and a name for the VIP resource, as well as the user that owns the application VIP resource. A VIP resource is typically owned by root because VIP related operations require root privileges.

To delete an application VIP, use the same script with the delete option. This option accepts the VIP name as a parameter. For example:

# Grid_home/bin/appvipcfg delete -vipname=appsVIP


After you have created the application VIP using this configuration script, you can view the VIP profile using the following command:


Grid_home/bin/crsctl status res appsVIP -p

Verify and, if required, modify the following parameters using the Grid_home/bin/crsctl modify res command.

The appvipcfg script assumes that the default ora.vip network resource (ora.net1.network) is used as the default. In addition, it is also assumes that a default app.appvip_net1.type is used for those purposes.

As the Oracle Database installation owner, start the VIP resource:

$ crsctl start resource appsVIP

Adding an Application VIP with Oracle Enterprise Manager

To add an application VIP with Oracle Enterprise Manager:

1.  Log into Oracle Enterprise Manager Database Control.
2.  Click the Cluster tab.
3.  Click Administration.
4.  Click Manage Resources.
5.  Enter a cluster administrator user name and password to   display the Manage Resources page.
6.  Click Add Application VIP.
7.  Enter a name for the VIP in the Name field.
8.  Enter a network number in the Network Number field.
9.  Enter an IP address for the VIP in the Internet Protocol Address field.
10. Enter root in the Primary User field. Oracle Enterprise Manager defaults to whatever user name you are logged in as.
11. Select Start the resource after creation if you want the VIP to start immediately.
12. Click Continue to display the Confirmation: Add VIP Resource page.
13. Enter root and the root password as the cluster credentials.
14. Click Continue to create the application VIP.

CLUSTER HA Components HA

Clusterware Components, Processes and Agents

Overview

·         Oracle Clusterware Version 11g Release 2 introduces the concept of the agent.

·         Agents are multi-threaded daemon programs that provide start, start, and cleanup and check actions for different resource types.

·         For example, the oraagent for crsd starts ASM, the oracle listener and starting the SCAN listener.

·         Agents can also receive, process and forward events to clients.

·         The standard agents in Oracle Clusterware 11g Release 2 are oraagent, orarootagent and cssdagent. Additionally there can be an application and script agents.

·         Agents create their own log files. These log files are contained in either ORA_CRS_HOME under a directory associated with the name of the agent.

There are a number of different processes that are associated with Oracle Clusterware. These processes are rolled up into several different Clusterware components. The following table lists the Components, associated processes and provide a description of the function of the component/process(es):

Component	Process	Description
Oracle High Availability Services (OHAS)	Ohasd	This process is responsible for starting the rest of the Oracle Clusterware stack on a given node. Ohasd is a brand new cluster startup framework in Oracle Clusterware 11g Release 2 that replaces the old init scripts.
Cluster Ready Service (CRS)	crsd	See the section titled CRS below for more information on this component and the crsd process.
Cluster Synchronization Service (CSS)	ocssd, cssdmonitor, cssdagent	See the section titled CSS below for more information on this component and the crsd process.
Event Manager (EVM)	evmd, evmlogger	Responsible for publishing Clusterware events.
Cluster Time Synchronization Service (CTSS)	octssd	Provides time synchronization services in an Oracle 11g Release 2 cluster.
Oracle Notification Service (ONS)	ons, enos	A publish-and-subscribe service responsible for communicating Fast Application Notification (FAN) events.
Oracle Agent	oraagent	The Oracle Agent is in conjunction with FAN to run scripts when specific Fan events occur.
Oracle Root Agent	orarootagent	This agent helps CRSD manage resources that are owned by root .
Grid Naming Sertvice (GNS)	gnsd	Provides gateway services between the multicast domain name service (which allows DNS requests) and external DNS services. GNS provides for name resolution within a cluster.
Grid Plug and Play (GPnP)	gpnpd	Supports Grid Plug and Play services, new in Oracle Clusterware 11g Release 2. GPnP provides services that allow you to easily add or remove nodes from a given cluster.
Multicast domain name service (mDNS)	mdnsd	This service services DNS requests.

Cluster Ready Services (CRS)

CRS is responsible for managing HA options within the cluster. The crsd process manages CRS operations. CRS manages two kinds of resources:

• Cluster resources
• Local resources 

A cluster resource is a resource that is cluster aware and is managed over the entire cluster via the crsctlcommand. Cluster resources are subject to cross-node switchover and failover. This means that a resource can be assigned to one or more nodes, but may be re-assigned to a different node (of failed over to a different node) on demand. Cluster resources are managed with the CRS daemon (crsd). The OCR is used by CRS to manage the resource.

A local resource runs on each node of the cluster. Examples of cluster resources are RAC instances and listeners. CRS can control these services, starting them, stopping them and restarting them in the event of a failure.

CSS

CSS is a service that is responsible for determining which nodes of the cluster are available to the cluster. CSS also supports other cluster processes by providing node membership information and locking services. The CSS uses the private interconnect for communications as well as the Clusterware voting disks. Through a combination of heartbeat messages over the interconnect and the voting disks CSS will determine the status of each node of the cluster. 

CSS is also responsible for interfacing with any third-party Clusterware vendors. In these configurations CSS will interface with the vendor Clusterware and maintain the node membership information.

The CSS service is critical to Clusterware operations as it fences the operations of the nodes of the cluster. For example, if the interconnect fails on a given node then the failed node will no longer be able to communicate with the rest of the cluster. Without CSS controlling the situation, the isolated node could cause severe issues on the cluster including corruption of database data. This is what is known as a split-brain condition.

To avoid split-brain conditions CSS sends heartbeat messages across the cluster interconnect. If a node fails (say the interconnect fails or the node freezes) then that node will no longer send heartbeat messages. The surviving nodes will detect that the heartbeat messages from the node are no longer being sent.  CSS then uses the voting disks to determine which node has gone offline. CSS will then work with Oracle Clusterware to evict the missing node from the cluster.

The CSS uses several different processes. Failure of these process will result in the restart of the cluster. The CSS process are:

• CSS daemon (ocssd) – Manages cluster node membership information. It’s also used in non-RAC installs to provide Group Services (GS). ASM uses GS to register itself and its disk groups.
• CSS Agent (cssdagent) – Monitors the cluster and provides fencing services (was oprocd daemon in previous versions). The CSS Agent is also responsible for monitoring vendor Clusterware.
• CSS Monitor (cssdmonitor) – This process monitors for node hangs, monitoris OCSSD processes for hangs and is also responsible for monitoring vendor Clusterware.

Clusterware Process Startup

Oracle Clusterware 11g Release 2 changes the way that Clusterware is started. In a Linux install, Clusterware is now started with one init script, init.ohasd which replaces a number of scripts that were previously used. The ohasd daemon sets off a cascade of processes as outlined in the following graphic:

Note: This graphic only summarizes the processes started by Oracle Clusterware.

You can control the startup or shutdown of the cluster via the crsctl command. For example, use crsctl start cluster to start the cluster and crsctl stop cluster to stop the cluster. You can also use the crsctl check cluster command to check on the status of the cluster. See the section titled “Managing Oracle Clusterware” for more information on crsctl and managing Oracle Clusterware.

Managing Resources in 11g RAC

Managing Resources


How to add resource using EM

This section includes the following topics:

• Registering Application Resources
• Starting Application Resources
• Relocating Applications and Application Resources
• Stopping Applications and Application Resources
• Displaying Clusterware Application and Application Resource Status Information

Registering Application Resources

Each application that you manage with Oracle Clusterware is stored as a resource in OCR. Use the crsctl add resource command to register applications in OCR. For example, enter the following command to register the Apache Web server application from the previous example:

$ crsctl add resource myApache -type cluster_resource

-attr "ACTION_SCRIPT=/opt/cluster/scripts/myapache.scr, PLACEMENT=restricted,

SERVER_POOLS=server_pool_list,CHECK_INTERVAL=30,RESTART_ATTEMPTS=2,

START_DEPENDENCIES=hard(appsvip),STOP_DEPENDENCIES=hard(appsvip)"

If you modify a resource, then update OCR by running the crsctl modifies resource command.

Starting Application Resources

Start and stop resources with the crsctl start resource and crsctl stop resource commands. Manually starting or stopping resources outside of Oracle Clusterware can invalidate the resource status. In addition, Oracle Clusterware may attempt to restart a resource on which you perform a manual stop operation.

To start an application resource that is registered with Oracle Clusterware, use the crsctl start resource command. For example:

$ crsctl start resource myApache

The command waits to receive a notification of success or failure from the action program each time the action program is called. Oracle Clusterware can start application resources if they have stopped due to exceeding their failure threshold values. You must register a resource using crsctl add resource before you can start it.

Running the crsctl start resource command on a resource sets the resource TARGET value to ONLINE. Oracle Clusterware attempts to change the state to match the TARGET by running the action program with the start action.

If a cluster server fails while you are starting a resource on that server, then check the state of the resource on the cluster by using the crsctl status resource command.

Relocating Applications and Application Resources

Use the crsctl relocate resource command to relocate applications and application resources. For example, to relocate the Apache Web server application to a server named rac2, run the following command:

# crsctl relocate resource myApache -n rac2

./crsctl relocate resource ora.scan1.vip -n rac1 -f

Each time that the action program is called, the crsctl relocate resource command waits for the duration specified by the value of the SCRIPT_TIMEOUT resource attribute to receive notification of success or failure from the action program. A relocation attempt fails if:

• The application has required resources that run on the initial server
• Applications that require the specified resource run on the initial server

To relocate an application and its required resources, use the -f option with the crsctl relocate resource command. Oracle Clusterware relocates or starts all resources that are required by the application regardless of their state.

Stopping Applications and Application Resources

Stop application resources with the crsctl stop resource command. The command sets the resource TARGET value to OFFLINE. Because Oracle Clusterware always attempts to match the state of a resource to its target, the Oracle Clusterware subsystem stops the application. The following example stops the Apache Web server:

# crsctl stop resource myApache

You cannot stop a resource if another resource has a hard stop dependency on it, unless you use the force (-f) option. If you use the crsctl stop resource resource_name -f command on a resource upon which other resources depend, and if those resources are running, then Oracle Clusterware stops the resource and all of the resources that depend on the resource that you are stopping.

Displaying Clusterware Application and Application Resource Status Information

To display status information about applications and resources that are on cluster servers, use the crsctl status resource command. The following example displays the status information for the Apache Web server application:

# crsctl status resource myApache

NAME=myApache

TYPE=cluster_resource

TARGET=ONLINE

STATE=ONLINE on server010

Other information this command returns includes the following:

• How many times the resource has been restarted
• How many times the resource has failed within the failure interval
• The maximum number of times that a resource can restart or fail
• The target state of the resource and the normal status information

Use the -f option with the crsctl status resource resource_name command to view full information of a specific resource.

Enter the following command to view information about all applications and resources in tabular format:

# crsctl status resource

Managing Automatic Restart of Oracle Clusterware Resources

You can prevent Oracle Clusterware from automatically restarting a resource by setting several resource attributes. You can also control how Oracle Clusterware manages the restart counters for your resources. In addition, you can customize the timeout values for the start, stop, and check actions that Oracle Clusterware performs on resources.

This section includes the following topics:

• Preventing Automatic Restarts
• Automatically Manage Restart Attempts Counter for Resources

Preventing Automatic Restarts

When a server restarts, Oracle Clusterware attempts to start the resources that run on the server as soon as the server starts. Resource startup might fail, however, if system components on which a resource depends, such as a volume manager or a file system, are not running. This is especially true if Oracle Clusterware does not manage the system components on which a resource depends. To manage automatic restarts, use the AUTO_START resource attribute to specify whether Oracle Clusterware should automatically start a resource when a server restarts.

Note:

Regardless of the value of the AUTO_START resource attribute for a resource, the resource can start if another resource has a hard or weak start dependency on it or if the resource has a pullup start dependency on another resource.

Automatically Manage Restart Attempts Counter for Resources

When a resource fails, Oracle Clusterware attempts to restart the resource the number of times specified in the RESTART_ATTEMPTS resource attributes, regardless of how often the resource fails. The crsd process maintains an internal counter to track how often Oracle Clusterware restarts a resource. The number of times Oracle Clusterware has attempted to restart a resource is reflected in the RESTART_COUNT resource attribute. Oracle Clusterware can automatically manage the restart attempts counter based on the stability of a resource. The UPTIME_THRESHOLD resource attribute determines the time period that a resource must remain online, after which the RESTART_COUNT attribute gets reset to 0. In addition, the RESTART_COUNT resource attribute gets reset to 0 if the resource is relocated or restarted by the user, or the resource fails over to another server.

Application placement policies

Application placement policies

A resource can be started on any server, subject to the placement policies, the resource start dependencies, and the availability of the action script on that server.

The PLACEMENT resource attribute determines how Oracle Clusterware selects a server on which to start a resource and where to relocate the resource after a server failure.

The HOSTING_MEMBERS and SERVER_POOLS attributes determine eligible servers to host a resource and the PLACEMENT attribute further refines the placement of resources.

The value of the PLACEMENT resource attribute determines how Oracle Clusterware places resources when they are added to the cluster or when a server fails. Together with either the HOSTING_MEMBERS or SERVER_POOLS attributes, you can configure how Oracle Clusterware places the resources in a cluster. When the value of the PLACEMENT attribute is:

·         balanced: Oracle Clusterware uses any online server for placement. Less loaded servers are preferred to servers with greater loads. To measure how loaded a server is, Oracle Clusterware uses the LOAD resource attribute of the resources that are in an ONLINE state on the server. Oracle Clusterware uses the sum total of the LOAD values to measure the current server load.

·         favored: If values are assigned to either the SERVER_POOLS or HOSTING_MEMBERS resource attribute, then Oracle Clusterware considers servers belonging to the member list in either attribute first. If no servers are available, then Oracle Clusterware places the resource on any other available server. If there are values for both the SERVER_POOLS and HOSTING_MEMBERS attributes, then the SERVER_POOLS attribute restricts the choices to the servers within the preference indicated by the value of HOSTING_MEMBERS.

·         restricted: Oracle Clusterware only considers servers that belong to server pools listed in the SEVER_POOLS resource attribute or servers listed in the HOSTING_MEMBERS resource attribute for resource placement. Only one of these resource attributes can have a value, otherwise it results in an error.

Unregistering Applications and Application Resources

To unregister a resource, use the crsctl delete resource command. You cannot unregister an application or resource that is ONLINE or required by another resource, unless you use the -force option. The following example unregisters the Apache Web server application:

$ crsctl delete resource myApache

Run the crsctl delete resource command as a clean-up step when a resource is no longer managed by Oracle Clusterware. Oracle recommends that you unregister any unnecessary resources.