RAC and Instance or Crash Recovery

RAC and Instance or Crash Recovery

When an instance fails and the failure is detected by another instance, the second instance performs the following recovery steps:

1.                  During the first phase of recovery, Global Enqueue Services (GES) remasters the enqueues.

2.                  The Global Cache Services (GCS) remasters its resources. The GCS processes remaster only those resources that lose their masters. During this time, all GCS resource requests and write requests are temporarily suspended. However, transactions can continue to modify data blocks as long as these transactions have already acquired the necessary resources.

3.                  After enqueues are reconfigured, one of the surviving instances can grab the Instance Recovery enqueue. Therefore, at the same time as GCS resources are remastered, SMON determines the set of blocks that need recovery. This set is called the recovery set. Because, with Cache Fusion, an instance ships the contents of its blocks to the requesting instance without writing the blocks to the disk, the on-disk version of the blocks may not contain the changes that are made by either instance. This implies that SMON needs to merge the content of all the online redo logs of each failed instance to determine the recovery set. This is because one failed thread might contain a hole in the redo that needs to be applied to a particular block. So, redo threads of failed instances cannot be applied serially. Also, redo threads of surviving instances are not needed for recovery because SMON could use past or current images of their corresponding buffer caches.

4.                  Buffer space for recovery is allocated and the resources that were identified in the previous reading of the redo logs are claimed as recovery resources. This is done to avoid other instances to access those resources.

5.                  All resources required for subsequent processing have been acquired and the Global Resource Directory (GRD) is now unfrozen. Any data blocks that are not in recovery can now be accessed. Note that the system is already partially available.
Then, assuming that there are past images or current images of blocks to be recovered in other caches in the cluster database, the most recent image is the starting point of recovery for these particular blocks. If neither the past image buffers nor the current buffer for a data block is in any of the surviving instances’ caches, then SMON performs a log merge of the failed instances. SMON recovers and writes each block identified in step 3, releasing the recovery resources immediately after block recovery so that more blocks become available as recovery proceeds. Refer to the section “Global Cache Coordination: Example” in this lesson for more information about past images.

6.                  After all blocks have been recovered and the recovery resources have been released, the system is again fully available.

In summary, the recovered database or the recovered portions of the database becomes available earlier, and before the completion of the entire recovery sequence. This makes the system available sooner and it makes recovery more scalable.

Note: The performance overhead of a log merge is proportional to the number of failed instances and to the size of the amount of redo written in the redo logs for each instance.

There are basically two types of failure in a RAC environment: instance and media. Instance failure involves the loss of one or more RAC instances, whether due to node failure or connectivity failure. Media failure involves the loss of one or more of the disk assets used to store the database files themselves.

If a RAC database undergoes instance failure, the first node still available that detects the failed instance or instances will perform instance recovery on all failed instances using the failed instances redo logs and the SMON process of the surviving instance. The redo logs for all RAC instances are located either on an OCFS shared disk asset or on a RAW file system that is visible to all the other RAC instances. This allows any other node to recover for a failed RAC node in the event of instance failure.

Recovery using redo logs allows committed transactions to be completed. Non-committed transactions are rolled back and their resources released.

There are experts with over a dozen years of working with Oracle databases that have yet to see an instance failure result in a non-recoverable situation with an Oracle database. Generally speaking, an instance failure in RAC or in normal Oracle requires no active participation from the DBA other than to restart the failed instance when the node becomes available once again.

If, for some reason, the recovering instance cannot see all of the datafiles accessed by the failed instance, an error will be written to the alert log. To verify that all datafiles are available, the ALTER SYSTEM CHECK DATAFILES command can be used to validate proper access.

Instance recovery involves nine distinct steps.  The Oracle manual only lists eight, but in this case, the actual instance failure has been included:

1. Normal RAC operation, all nodes are available.

2. One or more RAC instances fail.

3. Node failure is detected.

4. Global Cache Service (GCS) reconfigures to distribute resource management to the surviving instances.

5. The SMON process in the instance that first discovers the failed instance(s) reads the failed instance(s) redo logs to determine which blocks have to be recovered.

6. SMON issues requests for all of the blocks it needs to recover.  Once all blocks are made available to the SMON process doing the recovery, all other database blocks are available for normal processing.

7. Oracle performs roll forward recovery against the blocks, applying all redo log recorded transactions.

8. Once redo transactions are applied, all undo records are applied, which eliminates non-committed transactions.

9. Database is now fully available to surviving nodes.

Instance recovery is automatic, and other than the performance hit to surviving instances and the disconnection of users who were using the failed instance, recovery is invisible to the other instances. If RAC failover and transparent application failover (TAF) technologies are properly utilized, the only users that should see a problem are those with in-flight transactions. The following listing shows what the other instance sees in its alert log during a reconfiguration.

Sat Feb 15 16:39:09 2003
Reconfiguration started
List of nodes: 0,
 Global Resource Directory frozen
one node partition
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
 Resources and enqueues cleaned out
 Resources remastered 1977
 2381 GCS shadows traversed, 1 cancelled, 13 closed
 1026 GCS resources traversed, 0 cancelled
 3264 GCS resources on freelist, 4287 on array, 4287 allocated
 set master node info 
 Submitted all remote-enqueue requests
 Update rdomain variables
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 2381 GCS shadows traversed, 0 replayed, 13 unopened
 Submitted all GCS remote-cache requests
 0 write requests issued in 2368 GCS resources
 2 PIs marked suspect, 0 flush PI msgs
Sat Feb 15 16:39:10 2003
Reconfiguration complete
 Post SMON to start 1st pass IR
Sat Feb 15 16:39:10 2003
Instance recovery: looking for dead threads
Sat Feb 15 16:39:10 2003
Beginning instance recovery of 1 threads
Sat Feb 15 16:39:10 2003
Started first pass scan
Sat Feb 15 16:39:11 2003
Completed first pass scan
 208 redo blocks read, 6 data blocks need recovery
Sat Feb 15 16:39:11 2003
Started recovery at
 Thread 2: logseq 26, block 14, scn 0.0
Recovery of Online Redo Log: Thread 2 Group 4 Seq 26 Reading mem 0
  Mem# 0 errs 0: /oracle/oradata/ault_rac/ault_rac_raw_rdo_2_2.log
Recovery of Online Redo Log: Thread 2 Group 3 Seq 27 Reading mem 0
  Mem# 0 errs 0: /oracle/oradata/ault_rac/ault_rac_raw_rdo_2_1.log
Sat Feb 15 16:39:12 2003
Completed redo application
Sat Feb 15 16:39:12 2003
Ended recovery at
 Thread 2: logseq 27, block 185, scn 0.5479311
 6 data blocks read, 8 data blocks written, 208 redo blocks read
Ending instance recovery of 1 threads
SMON: about to recover undo segment 11
SMON: mark undo segment 11 as available

One word of caution, during testing for this listing, an instance could not be brought back up after failure, a rare occurrence. A kill -9 was done on the SMON process on AULTLINUX1, within the Linux/RAC/RAW environment.  AULTLINUX2 continued to operate and recovered the failed instance; however, an attempted restart of the instance on AULTLINUX1 yielded a Linux Error: 24: Too Many Files Open error. This was actually caused by something blocking the SPFILE link. Once the instance was pointed towards the proper SPFILE location during startup, it restarted with no problems.

RAC and Instance or Crash Recovery

RAC and Instance or Crash Recovery

When an instance fails and the failure is detected by another instance, the second instance performs the following recovery steps:

1.                  During the first phase of recovery, Global Enqueue Services (GES) remasters the enqueues.

2.                  The Global Cache Services (GCS) remasters its resources. The GCS processes remaster only those resources that lose their masters. During this time, all GCS resource requests and write requests are temporarily suspended. However, transactions can continue to modify data blocks as long as these transactions have already acquired the necessary resources.

3.                  After enqueues are reconfigured, one of the surviving instances can grab the Instance Recovery enqueue. Therefore, at the same time as GCS resources are remastered, SMON determines the set of blocks that need recovery. This set is called the recovery set. Because, with Cache Fusion, an instance ships the contents of its blocks to the requesting instance without writing the blocks to the disk, the on-disk version of the blocks may not contain the changes that are made by either instance. This implies that SMON needs to merge the content of all the online redo logs of each failed instance to determine the recovery set. This is because one failed thread might contain a hole in the redo that needs to be applied to a particular block. So, redo threads of failed instances cannot be applied serially. Also, redo threads of surviving instances are not needed for recovery because SMON could use past or current images of their corresponding buffer caches.

4.                  Buffer space for recovery is allocated and the resources that were identified in the previous reading of the redo logs are claimed as recovery resources. This is done to avoid other instances to access those resources.

5.                  All resources required for subsequent processing have been acquired and the Global Resource Directory (GRD) is now unfrozen. Any data blocks that are not in recovery can now be accessed. Note that the system is already partially available.
Then, assuming that there are past images or current images of blocks to be recovered in other caches in the cluster database, the most recent image is the starting point of recovery for these particular blocks. If neither the past image buffers nor the current buffer for a data block is in any of the surviving instances’ caches, then SMON performs a log merge of the failed instances. SMON recovers and writes each block identified in step 3, releasing the recovery resources immediately after block recovery so that more blocks become available as recovery proceeds. Refer to the section “Global Cache Coordination: Example” in this lesson for more information about past images.

6.                  After all blocks have been recovered and the recovery resources have been released, the system is again fully available.

In summary, the recovered database or the recovered portions of the database becomes available earlier, and before the completion of the entire recovery sequence. This makes the system available sooner and it makes recovery more scalable.

Note: The performance overhead of a log merge is proportional to the number of failed instances and to the size of the amount of redo written in the redo logs for each instance.

There are basically two types of failure in a RAC environment: instance and media. Instance failure involves the loss of one or more RAC instances, whether due to node failure or connectivity failure. Media failure involves the loss of one or more of the disk assets used to store the database files themselves.

If a RAC database undergoes instance failure, the first node still available that detects the failed instance or instances will perform instance recovery on all failed instances using the failed instances redo logs and the SMON process of the surviving instance. The redo logs for all RAC instances are located either on an OCFS shared disk asset or on a RAW file system that is visible to all the other RAC instances. This allows any other node to recover for a failed RAC node in the event of instance failure.

Recovery using redo logs allows committed transactions to be completed. Non-committed transactions are rolled back and their resources released.

There are experts with over a dozen years of working with Oracle databases that have yet to see an instance failure result in a non-recoverable situation with an Oracle database. Generally speaking, an instance failure in RAC or in normal Oracle requires no active participation from the DBA other than to restart the failed instance when the node becomes available once again.

If, for some reason, the recovering instance cannot see all of the datafiles accessed by the failed instance, an error will be written to the alert log. To verify that all datafiles are available, the ALTER SYSTEM CHECK DATAFILES command can be used to validate proper access.

Instance recovery involves nine distinct steps.  The Oracle manual only lists eight, but in this case, the actual instance failure has been included:

1. Normal RAC operation, all nodes are available.

2. One or more RAC instances fail.

3. Node failure is detected.

4. Global Cache Service (GCS) reconfigures to distribute resource management to the surviving instances.

5. The SMON process in the instance that first discovers the failed instance(s) reads the failed instance(s) redo logs to determine which blocks have to be recovered.

6. SMON issues requests for all of the blocks it needs to recover.  Once all blocks are made available to the SMON process doing the recovery, all other database blocks are available for normal processing.

7. Oracle performs roll forward recovery against the blocks, applying all redo log recorded transactions.

8. Once redo transactions are applied, all undo records are applied, which eliminates non-committed transactions.

9. Database is now fully available to surviving nodes.

Instance recovery is automatic, and other than the performance hit to surviving instances and the disconnection of users who were using the failed instance, recovery is invisible to the other instances. If RAC failover and transparent application failover (TAF) technologies are properly utilized, the only users that should see a problem are those with in-flight transactions. The following listing shows what the other instance sees in its alert log during a reconfiguration.

Sat Feb 15 16:39:09 2003
Reconfiguration started
List of nodes: 0,
 Global Resource Directory frozen
one node partition
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
 Resources and enqueues cleaned out
 Resources remastered 1977
 2381 GCS shadows traversed, 1 cancelled, 13 closed
 1026 GCS resources traversed, 0 cancelled
 3264 GCS resources on freelist, 4287 on array, 4287 allocated
 set master node info 
 Submitted all remote-enqueue requests
 Update rdomain variables
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 2381 GCS shadows traversed, 0 replayed, 13 unopened
 Submitted all GCS remote-cache requests
 0 write requests issued in 2368 GCS resources
 2 PIs marked suspect, 0 flush PI msgs
Sat Feb 15 16:39:10 2003
Reconfiguration complete
 Post SMON to start 1st pass IR
Sat Feb 15 16:39:10 2003
Instance recovery: looking for dead threads
Sat Feb 15 16:39:10 2003
Beginning instance recovery of 1 threads
Sat Feb 15 16:39:10 2003
Started first pass scan
Sat Feb 15 16:39:11 2003
Completed first pass scan
 208 redo blocks read, 6 data blocks need recovery
Sat Feb 15 16:39:11 2003
Started recovery at
 Thread 2: logseq 26, block 14, scn 0.0
Recovery of Online Redo Log: Thread 2 Group 4 Seq 26 Reading mem 0
  Mem# 0 errs 0: /oracle/oradata/ault_rac/ault_rac_raw_rdo_2_2.log
Recovery of Online Redo Log: Thread 2 Group 3 Seq 27 Reading mem 0
  Mem# 0 errs 0: /oracle/oradata/ault_rac/ault_rac_raw_rdo_2_1.log
Sat Feb 15 16:39:12 2003
Completed redo application
Sat Feb 15 16:39:12 2003
Ended recovery at
 Thread 2: logseq 27, block 185, scn 0.5479311
 6 data blocks read, 8 data blocks written, 208 redo blocks read
Ending instance recovery of 1 threads
SMON: about to recover undo segment 11
SMON: mark undo segment 11 as available

One word of caution, during testing for this listing, an instance could not be brought back up after failure, a rare occurrence. A kill -9 was done on the SMON process on AULTLINUX1, within the Linux/RAC/RAW environment.  AULTLINUX2 continued to operate and recovered the failed instance; however, an attempted restart of the instance on AULTLINUX1 yielded a Linux Error: 24: Too Many Files Open error. This was actually caused by something blocking the SPFILE link. Once the instance was pointed towards the proper SPFILE location during startup, it restarted with no problems.

RAC and Instance or Crash Recovery

RAC and Instance or Crash Recovery

When an instance fails and the failure is detected by another instance, the second instance performs the following recovery steps:

1.                  During the first phase of recovery, Global Enqueue Services (GES) remasters the enqueues.

2.                  The Global Cache Services (GCS) remasters its resources. The GCS processes remaster only those resources that lose their masters. During this time, all GCS resource requests and write requests are temporarily suspended. However, transactions can continue to modify data blocks as long as these transactions have already acquired the necessary resources.

3.                  After enqueues are reconfigured, one of the surviving instances can grab the Instance Recovery enqueue. Therefore, at the same time as GCS resources are remastered, SMON determines the set of blocks that need recovery. This set is called the recovery set. Because, with Cache Fusion, an instance ships the contents of its blocks to the requesting instance without writing the blocks to the disk, the on-disk version of the blocks may not contain the changes that are made by either instance. This implies that SMON needs to merge the content of all the online redo logs of each failed instance to determine the recovery set. This is because one failed thread might contain a hole in the redo that needs to be applied to a particular block. So, redo threads of failed instances cannot be applied serially. Also, redo threads of surviving instances are not needed for recovery because SMON could use past or current images of their corresponding buffer caches.

4.                  Buffer space for recovery is allocated and the resources that were identified in the previous reading of the redo logs are claimed as recovery resources. This is done to avoid other instances to access those resources.

5.                  All resources required for subsequent processing have been acquired and the Global Resource Directory (GRD) is now unfrozen. Any data blocks that are not in recovery can now be accessed. Note that the system is already partially available.
Then, assuming that there are past images or current images of blocks to be recovered in other caches in the cluster database, the most recent image is the starting point of recovery for these particular blocks. If neither the past image buffers nor the current buffer for a data block is in any of the surviving instances’ caches, then SMON performs a log merge of the failed instances. SMON recovers and writes each block identified in step 3, releasing the recovery resources immediately after block recovery so that more blocks become available as recovery proceeds. Refer to the section “Global Cache Coordination: Example” in this lesson for more information about past images.

6.                  After all blocks have been recovered and the recovery resources have been released, the system is again fully available.

In summary, the recovered database or the recovered portions of the database becomes available earlier, and before the completion of the entire recovery sequence. This makes the system available sooner and it makes recovery more scalable.

Note: The performance overhead of a log merge is proportional to the number of failed instances and to the size of the amount of redo written in the redo logs for each instance.

There are basically two types of failure in a RAC environment: instance and media. Instance failure involves the loss of one or more RAC instances, whether due to node failure or connectivity failure. Media failure involves the loss of one or more of the disk assets used to store the database files themselves.

If a RAC database undergoes instance failure, the first node still available that detects the failed instance or instances will perform instance recovery on all failed instances using the failed instances redo logs and the SMON process of the surviving instance. The redo logs for all RAC instances are located either on an OCFS shared disk asset or on a RAW file system that is visible to all the other RAC instances. This allows any other node to recover for a failed RAC node in the event of instance failure.

Recovery using redo logs allows committed transactions to be completed. Non-committed transactions are rolled back and their resources released.

There are experts with over a dozen years of working with Oracle databases that have yet to see an instance failure result in a non-recoverable situation with an Oracle database. Generally speaking, an instance failure in RAC or in normal Oracle requires no active participation from the DBA other than to restart the failed instance when the node becomes available once again.

If, for some reason, the recovering instance cannot see all of the datafiles accessed by the failed instance, an error will be written to the alert log. To verify that all datafiles are available, the ALTER SYSTEM CHECK DATAFILES command can be used to validate proper access.

Instance recovery involves nine distinct steps.  The Oracle manual only lists eight, but in this case, the actual instance failure has been included:

1. Normal RAC operation, all nodes are available.

2. One or more RAC instances fail.

3. Node failure is detected.

4. Global Cache Service (GCS) reconfigures to distribute resource management to the surviving instances.

5. The SMON process in the instance that first discovers the failed instance(s) reads the failed instance(s) redo logs to determine which blocks have to be recovered.

6. SMON issues requests for all of the blocks it needs to recover.  Once all blocks are made available to the SMON process doing the recovery, all other database blocks are available for normal processing.

7. Oracle performs roll forward recovery against the blocks, applying all redo log recorded transactions.

8. Once redo transactions are applied, all undo records are applied, which eliminates non-committed transactions.

9. Database is now fully available to surviving nodes.

Instance recovery is automatic, and other than the performance hit to surviving instances and the disconnection of users who were using the failed instance, recovery is invisible to the other instances. If RAC failover and transparent application failover (TAF) technologies are properly utilized, the only users that should see a problem are those with in-flight transactions. The following listing shows what the other instance sees in its alert log during a reconfiguration.

Sat Feb 15 16:39:09 2003
Reconfiguration started
List of nodes: 0,
 Global Resource Directory frozen
one node partition
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
 Resources and enqueues cleaned out
 Resources remastered 1977
 2381 GCS shadows traversed, 1 cancelled, 13 closed
 1026 GCS resources traversed, 0 cancelled
 3264 GCS resources on freelist, 4287 on array, 4287 allocated
 set master node info 
 Submitted all remote-enqueue requests
 Update rdomain variables
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 2381 GCS shadows traversed, 0 replayed, 13 unopened
 Submitted all GCS remote-cache requests
 0 write requests issued in 2368 GCS resources
 2 PIs marked suspect, 0 flush PI msgs
Sat Feb 15 16:39:10 2003
Reconfiguration complete
 Post SMON to start 1st pass IR
Sat Feb 15 16:39:10 2003
Instance recovery: looking for dead threads
Sat Feb 15 16:39:10 2003
Beginning instance recovery of 1 threads
Sat Feb 15 16:39:10 2003
Started first pass scan
Sat Feb 15 16:39:11 2003
Completed first pass scan
 208 redo blocks read, 6 data blocks need recovery
Sat Feb 15 16:39:11 2003
Started recovery at
 Thread 2: logseq 26, block 14, scn 0.0
Recovery of Online Redo Log: Thread 2 Group 4 Seq 26 Reading mem 0
  Mem# 0 errs 0: /oracle/oradata/ault_rac/ault_rac_raw_rdo_2_2.log
Recovery of Online Redo Log: Thread 2 Group 3 Seq 27 Reading mem 0
  Mem# 0 errs 0: /oracle/oradata/ault_rac/ault_rac_raw_rdo_2_1.log
Sat Feb 15 16:39:12 2003
Completed redo application
Sat Feb 15 16:39:12 2003
Ended recovery at
 Thread 2: logseq 27, block 185, scn 0.5479311
 6 data blocks read, 8 data blocks written, 208 redo blocks read
Ending instance recovery of 1 threads
SMON: about to recover undo segment 11
SMON: mark undo segment 11 as available

One word of caution, during testing for this listing, an instance could not be brought back up after failure, a rare occurrence. A kill -9 was done on the SMON process on AULTLINUX1, within the Linux/RAC/RAW environment.  AULTLINUX2 continued to operate and recovered the failed instance; however, an attempted restart of the instance on AULTLINUX1 yielded a Linux Error: 24: Too Many Files Open error. This was actually caused by something blocking the SPFILE link. Once the instance was pointed towards the proper SPFILE location during startup, it restarted with no problems.

RAC Performance Tuning Goal

• Optimize connection management, ensure that the middle tier and programs that connect to the database are efficient in connection management and do not log on or off repeatedly
• Tune the SQL using the available tools such as ADDM and SQL Tuning Advisor
• Ensure that applications use bind variables, cursor sharing was introduced to solve this problem
• Use packages and procedures (because they are compiled) in place of anonymous PL/SQL blocks and big SQL statements
• Use locally managed tablespaces and automatic segment space management to help performance and simplify database administration
• Use automatic undo management and temporary tablespace to simplify administration and increase performance
• Ensure you use large caching when using sequences, unless you cannot afford to lose sequence during a crash
• Avoid using DDL in production, it increases invalidations of the already parsed SQL statements and they need to be recompiled
• Partion tables and indexes to reduce index leaf contention (buffer busy global cr problems)
• Optimize contention on data blocks (hot spots) by avoiding small tables with too many rows in a block

Now we can review RAC specific best practices

• Consider using application partitioning (see below)
• Consider restricting DML-intensive users to using one instance, thus reducing cache contention
• Keep read-only tablespaces away from DML-intensive tablespaces, they only require minimum resources thus optimizing Cache Fusion performance
• Avoid auditing in RAC, this causes more shared library cache locks
• Use full tables scans sparingly, it causes the GCS to service lots of block requests, see table v$sysstat column "table scans (long tables)"
• if the application uses lots of logins, increase the value of sys.audsess$ sequence

Cache Coherency

Cache Coherency

GCS synchronizes global cache access, allowing only one instance at a time to modify the block. Thus, cache coherency is maintained in the RAC system by coordinating buffer caches located on separate instances.

GCS ensures that the data blocks cached in different cache buffers are maintained globally. That is why some people prefer to call cache fusion a ‘diskless cache coherency’ mechanism. This is true in a sense, because the previous Oracle parallel server version (OPS) utilized ‘forced disk writes’ to maintain cache coherency.

Global Cache Service

·         GCS is the main controlling process for cache fusion.

·         It tracks the location and status (mode and role) of the data blocks, as well as the access privileges of the various instances.

·         GCS guarantees data integrity by employing global access levels.

·         It maintains block modes for data blocks in the global role.

·         It is also responsible for block transfers between instances.

In a RAC system, users can connect with multiple instances to run database queries. Typically, users will be connected to different nodes but access the same set of data or data blocks. This situation demands that the data consistency, formerly confined to a single instance, be effectively extended to multiple instances. Therefore, buffer cache coherence from multiple instances must be maintained.

Instances require three main types of concurrency:

·         Concurrent reads on multiple instances — When users on two different instances need to read the same set of blocks.

·         Concurrent reads and writes on different instances — A user intends to read a data block that was recently modified, and the read can be for either the current version of the block, or for a read-consistent previous version.

·         Concurrent writes on different instances — When the same set of data blocks are modified by different users on different instances”

·         Cache Coherency demands that even though there are multiple instances (each with a separate db_cache_size data buffer region) in which data blocks can reside or brought in, block consistency must be maintained.

·         Oracle RAC achieves this by following the inter-instance block transfers through Cache Fusion mechanism.

·         The global cache services (GCS), which is implemented as a set of processes, organizes this facility.

·         GCS also ensures that only one instance modifies the block at any given time. Even when the same data block is cached in different instances at the same time, global consistency is maintained.

Data Block Writing Method

Oracle follows the concept of Dirty Block and Past Image of the block. Let’s understand what they are.

Whenever a server process changes or modifies a data block, it becomes a dirty block. Once a server process makes changes to the data block, the user may commit transactions, or transactions may not be committed for quite some time. In either case, the dirty block is not immediately written back to disk.

Writing dirty blocks to disk takes place under the following two conditions:

·         When a server process cannot find a clean, reusable buffer after scanning a threshold number of buffers, then the database writer process writes the dirty blocks to disk.

·         When the checkpoint takes place the database writer process writes the dirty blocks to disk”

As we are aware, a typical data block is not written to the disk immediately, even after it becomes dirty as the result of an update.

When the same dirty data block is requested by another instance for write or read purposes, an image of the block is created at the owning instance, and only that block is shipped to the requesting instance. This backup image of the block is called the past image (PI) and is kept in memory.

In the event of instance failure, Oracle can reconstruct the current version of the block by reading the PIs from RAM. It is also possible to have more than one past image in the memory depending on how many times the data block was requested in the dirty stage. The process of writing the blocks back to the I/O device (disk storage unit) depends on the checkpoint schedule defined by the DBA for the RAC cluster. Once the checkpoint interval is reached, Oracle’s Database Writer (DBWR) process initiates an asynchronous write of the dirty blocks to disk.

When the write takes place, a message is sent across Cache Fusion to change the status for the block in the other instances and the past images (PI), on all other instances are invalidated and discarded.

For more details, refer to Oracle Metalink Document Note # 139436.1 titled, “Understanding 9i Real Application Clusters Cache Fusion.”

Internal Lock Messaging in RAC

Remember, Oracle uses a lock escalation mechanism to maintain cache coherency. There can only be one block buffered in the “xcur” exclusive state in the cluster at any one time and to modify a block, each instance must assign an xcur state to the buffer containing the block. 

For example, if another instance requests reading the same block in its most current version, then oracle sends a message to change the access mode from exclusive to shared, sends the block to the requesting instance and keeps a Prior Image (PI) buffer if the buffer contained a dirty (changed) block. It then sends a “current read” version of the block to the requesting instance. The original instance keeps a copy in current mode, but the overall status of the block becomes global. Again, there can be multiple copies of the shared current (scurmode) cached at any time.

In early versions of Oracle OPS, one master instance kept track of the lock status, so if the master instance crashed, the entire OPS system went down. Obviously, this was a serious shortcoming, remedied in RAC. In later versions of OPS and RAC, only the uncommitted transactions on the instance that goes down are lost. The other instances stay active.

In RAC there is still a master node, but while the first node to start-up becomes the “master” node, it is strictly a bookkeeping method, and there are no repercussions to the cluster if the master node dies. The Cache Fusion mechanisms for Global Caching Service (GCS) and Global Enqueue Service (GES) are global resources, running on all nodes in the cluster, serving to maintain copies of the global dictionary.

Now that we understand the RAC block updating process, we are ready to move even deeper into RAC internals. Our next installment will examine RAC invalidation mechanisms.

Cache Fusion

Prior to Oracle 9, network-clustered Oracle databases used a storage device as the data-transfer medium (meaning that one node would write a data block to disk and another node would read that data from the same disk), which had the inherent disadvantage of lackluster performance.

Oracle 9i addressed this issue: RAC uses a dedicated network connection for communications internal to the cluster.

Since all computers/instances in a RAC access the same database, the overall system must guarantee the coordination of data changes on different computers such that whenever a computer queries data, it receives the current version — even if another computer recently modified that data. Oracle RAC refers to this functionality as Cache Fusion.

Cache Fusion involves the ability of Oracle RAC to "fuse" the in-memory data cached physically separately on each computer into a single, global cache.