Making Application Highly Available with Oracle Clusterware

Making Application Highly Available with Oracle Clusterware

Oracle Clusterware is a portable cluster infrastructure that provides HA services to RAC databases and other applications. Oracle Clusterware makes applications highly available by monitoring the health of the applications, by restarting applications on failure, and by relocating applications to another cluster node when the currently used node fails or when the application can no longer run in the current node. In the case of node failure, certain types of protected applications, such as a RAC database instance, may not be failed over to the surviving nodes.

A cluster is a collection of two or more nodes where the nodes share a common pool of storage used by the Oracle Clusterware system files (the Oracle Cluster Registry [OCR] and the voting disk), a common network interconnect, and a common operating system.

Oracle Clusterware monitors all protected applications periodically, and based on the defined failover policy, it can restart them either on the same node or relocate them to another node, or it can decide to not restart them at all.

Objectives

• Describe the high availability components of Oracle Clusterware
• Contrast policy-managed and administration-managed databases
• Describe the functionality of server pools
• Describe application-placement policies
• Create an application Virtual IP (VIP)
• Manage application resources

Topics

1. Managing Oracle Notification Server with srvctl HA App with clusterware
2. Managing Resources with crsctl
   
3. Adding Resources by Using crsctl HA App with clusterware
   
4. Defining Resource Dependencies HA App with clusterware
5. To add a new resource type to Oracle Clusterware by using Oracle Enterprise Manager HA App with clusterware
   
6. Clusterware Resource Modeling HA App with clusterware
   
7. Server Pools HA App with clusterware
   
8. Creating Server Pools with srvctl and crsctl HA App with clusterware
9. Resource Management Options
   
10. Oracle Clusterware HA Components
    
11. Oracle Clusterware High Availability (HA)
12. Creating an Application VIP Managed by Oracle Clusterware

Refer the links below:

Making Application HA with oracle clusterware
High Availability Architectures and Solutions

Managing Oracle Notification Server with srvctl

Managing Oracle Notification Server with srvctl

To determine the current ONS configuration:

$ srvctl config nodeapps -s

ONS daemon exists. Local port 6100, remote port 6200

 
To add Oracle Notification Server:

$ srvctl add nodeapps -l 6100 -r 6200

 

To start or stop Oracle Notification Server:

$ srvctl start|stop nodeapps

Managing Resources with crsctl

Start or stop resources with the crsctl start|stop resource command:


# crsctl stop resource myApache

# crsctl start resource myApache
 

Use the crsctl relocate resource command to relocate applications and application resources:

# crsctl relocate resource myApache -n host02

Adding Resources by Using crsctl

Resources can be registered in Oracle Clusterware by using the crsctl add resource command.

To add the Apache web server as:

An administrator-managed resource:

# crsctl add resource myApache -type cluster_resource -attr "ACTION_SCRIPT='/u01/ogi/scripts/myapache.scr', PLACEMENT=’restricted’, HOSTING_MEMBERS=’host01 host02’, CHECK_INTERVAL=’30’,START_DEPENDENCIES=’hard(appsvip)’,
STOP_DEPENDENCIES=’hard(appsvip)’, RESTART_ATTEMPTS=’2’,"

A policy-managed resource:

# crsctl add resource myApache -type cluster_resource -attr "ACTION_SCRIPT=’/u01/ogi/scripts/myapache.scr’, PLACEMENT=’restricted’, SERVER_POOLS=’myServerPool’, CHECK_INTERVAL=’30’, START_DEPENDENCIES=’hard(appsvip)’, STOP_DEPENDENCIES=’hard(appsvip)’,RESTART_ATTEMPTS=’2’,"

To delete the resource:

# crsctl delete resource myApache

Defining Resource Dependencies

Defining Resource Dependencies

Click Dependencies to configure start and stop dependencies between resources for the new resource type. Two or more resources are dependent when they either depend on each other or affect other resources. For example, when Clusterware attempts to start a resource, it is necessary for any resources on which the initial resource depends to be online and in the same location. If Oracle Clusterware cannot bring the resources online, the initial (dependent) resource cannot be brought online either. Resource dependencies are separated into start and stop categories. This separation improves and expands the start and stop dependencies between resources and resource types.

Oracle Clusterware considers start dependencies between resources whenever a resource state changes from OFFLINE to ONLINE. Start dependencies displayed in EM include:

• Hard: This defines a hard start dependency for a resource if another resource must be ONLINE before the dependent resource can start.
• Weak: This means that an attempt is made to start the resource on which the resource in question is dependent on if it is not ONLINE.
• Attraction: This indicates that Clusterware will attempt to start a resource on the same node on which the resource it is dependent on is running.Pullup: When you specify the pullup start dependency for a resource, this resource starts as a result of starting the named resources.
• Dispersion: This indicates that the resource will not be located on the same server as dependent resources, if possible.
• Stop Dependencies

Oracle Clusterware considers stop dependencies between resources whenever a resource state changes from ONLINE to OFFLINE.

Hard means that if a resource has a hard stop dependency on another resource, then the dependent resource must be stopped when the second transitions into the INTERMEDIATE, OFFLINE, or UNKNOWN state. The two resources may attempt to start or relocate to another server, depending upon how they are configured.

Clicking the flashlight icon to the right of the “Dependent On Resource” field brings up a list of registered resources to select from. When you have finished, return to the General folder tab and click the Submit button.

Reference link

To add a new resource type to Oracle Clusterware by using Oracle Enterprise Manager

How to add resources

How to add resources using command

• Log in to Oracle Enterprise Manager Database Control and click the Cluster tab and then the Administration subfolder tab. Then click Add Resource Type.
• Enter a cluster administrator username and password to display the Add Resource Type page.
• Enter a name for the resource type in the Name field.
• Select either cluster_resource or local_resource as the base resource type from the Base Resource Type drop-down list. Generally, select the cluster_resource type for resources that can reside on any server in a cluster. Select the local_resource type for resources that must be present on each server of a cluster, by definition, such as VIPs, ASM instances, and network resources. Clicking the View button allows you to view the chosen resource type’s characteristics, parameters, and dependencies.
• The optional parameters in the Placement section define where in a cluster Oracle Clusterware places the resource. Placement characteristics include:
• Cardinality: This indicates the number of nodes the resource should run on simultaneously.
• Degree: This indicates the number of times a resource can run simultaneously on a single node.
• Server Pool: This allows the resource to be assigned to a server pool if server pools have been defined. This would indicate that resources configured using this type will be policy managed.
• Hosting Members: This creates an association between a resource and a group of hosts. This would indicate that resources configured using this type will be administrator managed.
• Placement Policy: A balanced policy allows the resource to be placed on any online node.
• Load: Clusterware interprets this value along with that of the PLACEMENT value. When the value of PLACEMENT is balanced, the value of LOAD determines where best to place a resource.
• Active placement: When set to 1, Clusterware uses this attribute to reevaluate the placement of a resource during addition or restart of a cluster server.
• In the Action Program section, select from the Action Program drop-down list whether Oracle Clusterware calls an action script, an agent file, or both to manage the resource.

 




Resource Type Parameters

To further configure the resource type, click the Parameters tab. Resource parameters can be categorized as start parameters, stop parameters, or status parameters, as shown in the slide. The configurable parameters here include:

• Auto Start: Indicates whether Oracle Clusterware automatically starts a resource after a cluster server restart
• Start Attempts: Represents the number of times that Oracle Clusterware attempts to start a resource on the current server before attempting to relocate it
• Start Timeout: Is the maximum time in seconds in which a start action can run. Oracle Clusterware returns an error message if the action does not complete within the time specified.
• Uptime Threshold: Represents the length of time that a resource must be up before Oracle Clusterware considers the resource to be stable
• Stop Timeout: Is the maximum time (in seconds) in which a stop action can run
• Check Interval: Is the interval in seconds between repeated executions of the check action
• Script Timeout: Is the maximum time (in seconds) for an action script to run. The timeout applies to all actions (start, stop, check, and clean).
• Failure Interval: Is the interval, in seconds, during which Oracle Clusterware applies the Failure Threshold attribute. If the value is zero (0), tracking of failures is disabled.

Refer this link

Clusterware Resource Modeling

Clusterware Resource Modeling

• When an application is registered, you define how Clusterware manages it by defining its attributes.
• The registration information includes an action script or action program that Clusterware calls to start, stop, check, and clean up the application.
• An action script is a shell script that a generic script agent provided by Oracle Clusterware calls.
• An application-specific agent is usually a C or C++ program that calls Clusterware APIs directly.
• Oracle Clusterware 11g R2 includes two agents:
• Script agent: CRS_HOME/bin/scriptagent
• Application agent: CRS_HOME/bin/appagent

Resource Types

• Oracle Clusterware uses resource types to organize resources employing similar attributes.
• Benefits from the use of resource types include:
• You need to manage only necessary resource attributes.
• You can manage all resources based on the resource type.
• All resources registered with Oracle Clusterware must be associated with a resource type.
• There are two resource types predefined in Oracle Clusterware:
• Local resource
• Cluster resource
• You can define custom resource types with Enterprise Manager.

Server Pools

Server Pools

Server pools are logical divisions of a cluster into pools of servers or nodes. They:

• Distribute a uniform workload over several servers in the cluster
• Are allocated to host databases or other resources
• Are managed using the crsctl and srvctl commands
• Support parent-child relationships among server pools
• Top-level pools are mutually exclusive.
• Include two built-in server pools at Oracle Clusterware installation:
• FREE: For servers that are not assigned to other pools
• GENERIC: For administrator-managed fixed configuration and pre-11g Release 2 databases

Server Pool Attributes

Server pool attributes can be specified when you create a server pool, or they can be modified after a server pool has already been created. The only required attribute is NAME. The optional attributes include:

• ACL: Defines the owner of the server pool and what privileges are granted to various operating system users and groups. The value of this attribute is populated at the time a server pool is created based on the identity of the process creating the server pool, unless explicitly overridden. It uses a string in the format of: owner:user:rwx,pgrp:group:rwx,other::r— to specify allowed privileges for the owner, group, and other users. The privileges allowed for a server pool are read-only (r), the ability to modify attributes or delete the server pool (w), and the ability to assign resources to the pool (x).

• ACTIVE_SERVERS: Is a space-delimited list of servers that are currently assigned to the server pool. This attribute is automatically managed by Oracle Clusterware.
• EXCLUSIVE_POOLS: Governs whether servers assigned to this server pool are shared with other server pools. This attribute is a string value containing any arbitrary string. Any other server pool that has the same value for this string is mutually exclusive with this server pool. 
• IMPORTANCE: Determines the relative importance of the server pool compared to other server pools, with 0 denoting the lowest level of importance and 1000 the highest level of importance. The default value is 0.
• MAX_SIZE: Determines the maximum number of servers a server pool can contain. A value of –1 for this attribute spans the entire cluster and is the default value.
• MIN_SIZE: Determines the minimum number of servers a server pool can contain. The value of this attribute does not set a hard limit. It governs the priority for server assignment. The default value is 0.
• PARENT_POOLS: Allows the creating of nested server pools. Server pools listed in this attribute are referred to as parent server pools. Multiple parent server pools may be specified by using a comma-delimited list of server pool names.
• SERVER_NAMES: Lists the candidate node names upon which servers reside that may be associated with a server pool. If this attribute is empty, Oracle Clusterware assumes that any server may be assigned to any server pool, to the extent allowed by other attributes, such as PARENT_POOLS.

Note: All attributes of the GENERIC server pool are read-only and cannot be modified. For the FREE server pool, only the IMPORTANCE and ACL attributes can be edited.

Use the crsctl status server command to check the server status:
# crsctl status server -f

Introduction to 11gR2 RAC ServerPools

 

Creating Server Pools with srvctl and crsctl

Use the crsctl utility or the srvctl utility to create additional server pools.

• Specifying attributes on the command line:

$ crsctl add serverpool SP1 –attr "MIN_SIZE=2, MAX_SIZE=5, IMPORTANCE=3"

$ srvctl add srvpool –g SP1 –l 2 –u 5 –i 3 -n "server1,server2"

• Specifying attributes using a text file to supply them:

$ crsctl add serverpool SP1 –file /usr/local/bin/SP1_attributes.txt



MIN_SIZE (-l), MAX_SIZE (-u), IMPORTANCE (-i), and SERVER_NAMES (-n)
 
Note: New pools can be created only after the Generic pool has been deleted.



Managing Server Pools with srvctl and crsctl

To delete server pools:

$ crsctl delete serverpool SP1

$ srvctl remove srvpool –g SP1


To modify server pools:

$ crsctl modify serverpool SP2 –attr "MIN_SIZE=4, MAX_SIZE=8, IMPORTANCE=7"

$ srvctl modify srvpool -g SP2 –l 4 –u 8 –i 7

Resource Management Options

Resource Management Options

There are two styles of configuring resource management for a RAC database:

• administrator-managed configuration
• policy-managed configuration

Each management style uses the concept of server pools, which are a logical division of a cluster into pools of servers.

Administrator-managed configuration 

• is the management strategy used in previous releases. 
• For this style, the database administrators define on which servers a database resource should run and place resources manually as needed. 
• It uses a subpool of the built-in GENERIC server pool. 
• This style is best suited for a small number of nodes due to the manual placement requirements.

Policy-managed configuration 

• controls the placement of all resources by using server pools.
• Database administrators specify in which server pool (excluding GENERIC or FREE ) the database resource will run. 
• Oracle Clusterware is then responsible for placing the database resource on a specific server. 
• It eliminates physically assigning resources to particular nodes in the cluster. 
• This style is workload management compliant and best for a large number of nodes.

How to create Policy-managed RAC Database

Managing Resources with crsctl

Oracle Clusterware HA Components

 Oracle Clusterware HA Components

• Resource
• Action Program 
• Privileges
• Resource Dependency
• Application VIP
• OCR

Several components work together in building a Highly Available (HA) framework for applications using Oracle Clusterware.

A resource is an entity that Oracle Clusterware manages for HA such as an application. Resources for HA are defined with an application profile that describes attributes and policies for the application.

The application profile also identifies the action program, which is a program or script responsible for providing logic to start, stop, and check the status of a resource. The application profile also defines the failure policies for an application. The Oracle Clusterware software runs with root or administrator rights.

Privileges enable Oracle Clusterware to control the components of an application to include allowing the application to run under the context of a different user from that which Cluster Ready Services (CRS) runs under.

Resources can have a dependency on other resources for operation. For example, a database resource may depend on a storage resource to be running.

An application Virtual IP (VIP) is a VIP that can fail over to other nodes if policies allow it and is one example of a typical application dependency. The application VIP is a resource.

All the information about a resource is stored in the Oracle Clusterware OCR configuration file and is available to each node in the cluster.

Oracle Clusterware High Availability (HA)

Oracle Clusterware provides HA services to Real Application Clusters (RAC) databases and other applications. Oracle Clusterware makes applications highly available by monitoring the health of the applications, by restarting applications on failure, and by relocating applications to another cluster node when the currently used node fails or when the application can no longer run in the current node.

• Oracle Clusterware monitors all protected applications periodically.
• Based on the defined failover policy, Oracle Clusterware can restart failed applications on the same node or relocate them to another node.
• It can protect Oracle-based as well as non-Oracle-based applications.

A cluster is a collection of two or more nodes where the nodes share a common pool of storage used by the Oracle Clusterware system files (the Oracle Cluster Registry [OCR] and the voting disk), a common network interconnect, and a common operating system.

Oracle Clusterware monitors all protected applications periodically, and based on the defined failover policy, it can restart them either on the same node or relocate them to another node, or it can decide to not restart them at all.

Creating an Application VIP Managed by Oracle Clusterware

Creating an Application VIP Managed by Oracle Clusterware

If clients of an application access the application through a network, and the placement policy for the application allows it to fail over to another node, then you must register a virtual internet protocol address (VIP) on which the application depends. An application VIP is a cluster resource that Oracle Clusterware manages (Oracle Clusterware provides a standard VIP agent for application VIPs). 

 

You should base any new application VIPs on this VIP type to ensure that your system experiences consistent behavior among all of the VIPs that you deploy in your cluster.

While you can add a VIP in the same way that you can add any other resource that Oracle Clusterware manages, Oracle recommends using the script Grid_home/bin/appvipcfg to create or delete an application VIP.

To create an application VIP, use the following syntax:


appvipcfg create -network=network_number -ip=ip_address -vipname=vip_name -user=user_name [-group=group_name] [-failback=0 | 1]


To delete an application VIP, use the following syntax:


appvipcfg delete -vipname=vip_name

Where network_number is the number of the network, ip_address is the IP address, vip_name is the name of the VIP, user_name is the name of the user who installed Oracle Database, and group_name is the name of the group. The default value of the -failback option is 0.

 

If you set the option to 1, then the VIP (and therefore any resources that depend on VIP) fails back to the original node when it becomes available again.

For example, as root, run the following command:


# Grid_home/bin/appvipcfg create -network=1 -ip=148.87.58.196 -vipname=appsVIP -user=root

The script only requires a network number (default is 1), the IP address, and a name for the VIP resource, as well as the user that owns the application VIP resource. A VIP resource is typically owned by root because VIP related operations require root privileges.

To delete an application VIP, use the same script with the delete option. This option accepts the VIP name as a parameter. For example:

# Grid_home/bin/appvipcfg delete -vipname=appsVIP


After you have created the application VIP using this configuration script, you can view the VIP profile using the following command:


Grid_home/bin/crsctl status res appsVIP -p

Verify and, if required, modify the following parameters using the Grid_home/bin/crsctl modify res command.

The appvipcfg script assumes that the default ora.vip network resource (ora.net1.network) is used as the default. In addition, it is also assumes that a default app.appvip_net1.type is used for those purposes.

As the Oracle Database installation owner, start the VIP resource:

$ crsctl start resource appsVIP

Adding an Application VIP with Oracle Enterprise Manager

To add an application VIP with Oracle Enterprise Manager:

1.  Log into Oracle Enterprise Manager Database Control.
2.  Click the Cluster tab.
3.  Click Administration.
4.  Click Manage Resources.
5.  Enter a cluster administrator user name and password to   display the Manage Resources page.
6.  Click Add Application VIP.
7.  Enter a name for the VIP in the Name field.
8.  Enter a network number in the Network Number field.
9.  Enter an IP address for the VIP in the Internet Protocol Address field.
10. Enter root in the Primary User field. Oracle Enterprise Manager defaults to whatever user name you are logged in as.
11. Select Start the resource after creation if you want the VIP to start immediately.
12. Click Continue to display the Confirmation: Add VIP Resource page.
13. Enter root and the root password as the cluster credentials.
14. Click Continue to create the application VIP.