Administering ASM Cluster File Systems

Administering ASM Cluster File Systems

Oracle Automatic Storage Management Cluster File System (Oracle ACFS) is a multi-platform, scalable file system, and storage management technology that extends Oracle Automatic Storage Management (Oracle ASM) functionality to support customer files maintained outside of Oracle Database. Oracle ACFS supports many database and application files, including executables, database trace files, database alert logs, application reports, BFILEs, and configuration files. Other supported files are video, audio, text, images, engineering drawings, and other general-purpose application file data.

The ASM feature of Oracle database has been extended in Oracle Grid Infrastructure 11g Release 2 to include support for a general-purpose cluster file system, the ASM Cluster File System (ACFS). To understand the operation of this feature, some terminology needs to be defined and explained. At the operating system (OS) level, the ASM instance provides the disk group, which is a logical container for physical disk space. The disk group can hold ASM database files and ASM dynamic volume files. The ASM Dynamic Volume Manager (ADVM) presents the volume device file to the operating system as a block device. The mkfs utility can be used to create an ASM file system in the volume device file.

Four OS kernel modules loaded in the OS provide the data service. On Linux, they are: oracleasm, the ASM module; oracleadvm, the ASM dynamic volume manager module; oracleoks, the kernel services module; and oracleacfs, the ASM file system module. These modules provide the ASM Cluster File System, ACFS snapshots, the ADVM, and cluster services. The ASM volumes are presented to the OS as a device file at /dev/asm/<volume name>-<number>.

The volume device file appears as another ASM file to the ASM Instance and asmcmd utility. The ASM layers are transparent to the OS file system commands. Only the files and directories created in ACFS and the ACFS snapshots are visible to the OS file system commands. Other file system types such as ext3 and NTFS may be created in an ADMV volume using the mkfs command on Linux and advmutil commands on Windows.

Objectives

• Administer ASM Dynamic Volume Manager
• Manage ASM volumes
  
• Implement ASM Cluster File System
• Manage ASM Cluster File System (ACFS)
• Use ACFS Snapshots
• Use command-line tools to manage ACFS
  
  

Topics

1. ACFS and ADVM Architecture Overview
2. ASM Cluster File System
3. ADVM Processes
4. Striping Inside the Volume
5. Creating an ACFS Volume
6. Extending ASMCMD for Dynamic Volumes 
7. ACFS Snapshots
8. ACFS Replication

Refer the links below:

Introduction to Oracle ACFS
Oracle ACFS Advanced Topics
Administrating ACFS

ACFS Replication

Oracle ACFS replication enables replication of Oracle ACFS file systems across the network to a remote site, providing disaster-recovery capability for the file system. Oracle ACFS replication can only be configured for Oracle RAC systems. The source Oracle ACFS file system of an Oracle ACFS replication is referred to as a primary file system. The target Oracle ACFS file system of an Oracle ACFS replication is referred to as a standby file system.

A site can host both primary and standby file systems. For example, if there are cluster sites A and B, a primary file system hosted at site A can be replicated to a standby file system at site B. Also, a primary file system hosted at site B can be replicated to a standby file system at site A. However, an ACFS file system cannot be used as a primary and a standby file system.

Oracle ACFS replication captures file system changes written to disk for a primary file system and records the changes in files called replication logs. These logs are transported to the site hosting the associated standby file system where background processes read the logs and apply the changes recorded in the logs to the standby file system. After the changes recorded in a replication log have been successfully applied to the standby file system, the replication log is deleted from the sites hosting the primary and standby file systems.

It is critical that there is enough disk space available on both sites hosting the primary and the standby file systems to contain the replication logs.

ACFS Replication Requirements

Before using replication on a file system, ensure that you have checked the following:

• There is sufficient network bandwidth to support replication between the primary and standby file systems.
• The configuration of the sites hosting the primary and standby file systems allow the standby file system to keep up with the rate of change on the primary file system.
• The standby file system has sufficient capacity to manage the replication logs.
• There is sufficient storage capacity to hold excess replication logs that might collect on the primary and the standby file systems when the standby file system cannot process replication logs quickly. For example, this situation can occur during network problems or maintenance on the site hosting the standby file system.
• The primary file system must have a minimum size of 4 GB for each node that is mounting the file system. The standby file system must have a minimum size of 4 GB and should be sized appropriately for the amount of data being replicated and the space necessary for the replication logs sent from the primary file system.

Before replicating an ACFS file system, a replication configuration must be established that identifies information such as the site hosting the primary and standby file systems, the file system to be replicated, mount point of the file system, and a list of tags if desired.

To use Oracle ACFS replication functionality on Linux, the disk group compatibility attributes for ASM and ADVM must be set to 11.2.0.2 or higher for the disk groups that contain the primary and standby file systems. To configure replication and manage replicated Oracle ACFS file systems, use the acfsutil repl command-line functions.

Managing ACFS Replication

The basic steps for managing ACFS replication are:

1. Determine the storage capacity necessary for replication on the sites hosting the primary and standby file systems.

2. Set up usernames, service names, and tags.

SQL> CREATE USER primary_admin IDENTIFIED BY primary_passwd;

SQL> GRANT sysasm,sysdba TO primary_admin;

primary_repl_site=(DESCRIPTION=

(ADDRESS=(PROTOCOL=tcp)(HOST=primary1.example.com)(PORT=1521))

(ADDRESS=(PROTOCOL=tcp)(HOST=primary2.example.com)(PORT=1521))

(CONNECT_DATA=(SERVICE_NAME=primary_service)))

standby_repl_site=(DESCRIPTION= ...

3. Configure the site hosting the standby file system.

$ /sbin/acfsutil repl init standby \

-p primary_admin/primary_passwd@primary_repl_site \

-c standby_repl_service /standby/repl_data

4. Configure the site hosting the primary file system.

$ /sbin/acfsutil repl init primary \

-s standby_admin/standby_passwd@standby_repl_site \

-m /standby/repl_data -c primary_repl_service \

/acfsmounts/repl_data

5. Monitor information about replication on the file system.

6. Manage replication background processes.



ACFS Backups

• An ACFS file system may be backed up using:
• Standard OS file system backup tools
• Oracle Secure Backup
• Third-party backup tools
• ACFS snapshots present a stable point-in-time view.

ACFS Performance

ACFS performance benefits from:

• Distribution and load balancing of ASM file segments
• ACFS file extents distributed across ASM file segments
• User and metadata caching
• In-memory updates of transaction logs

ACFS Snapshots

An Oracle ACFS snapshot is an online, read-only, point-in-time copy of an Oracle ACFS file system.

The snapshot copy is initially sparse and merely references the storage allocation information maintained by the file system.

Before an Oracle ACFS file extent is modified or deleted, its current value is copied to the snapshot using a copy-on-write (COW) technique to maintain the snapshot’s point-in-time view of the file system.

ACFS snapshots provide a low-cost way to preserve the state of a volume at a point in time. When a snapshot is created, a set of metadata about the volume is created in the same volume.

This metadata includes the directory structure and the names of all the files.

The snapshot consists of pointers to blocks in the volume that have not changed since the snapshot was created and the actual blocks if the file has changed.

After the snapshot is created, data blocks are written to the snapshot as files change, but only the original version of changed blocks are written. The snapshot allows you to recover deleted files, revert files to the version at the time of the snapshot, and back up a consistent file set by taking a backup of the snapshot.

Can be used to:

• Revert to a version in the snapshots
• Recover a deleted file
• Back up a consistent data set
• Are limited to 63 snapshot views per volume

Managing ACFS Snapshots

Create snapshots:

$ acfsutil snap create snapshot_2 /u01/app/oracle/acfsdata/testvol
 

Delete snapshots:

$ acfsutil snap delete snapshot_2 /u01/app/oracle/acfsdata/testvol
 

View file system information, including snapshots:

$ acfsutil info fs <mount point> ls –l <mount_point>/.ACFS/snaps

Creating an ACFS Volume

The COMPATIBLE.ADVM attribute of the disk group must be 11.2.0 or higher before you can create an ADVM volume in the disk group. You can set the attribute with the command:

asmcmd setattr -G DATA compatibility.advm 11.2.0.0.0



Create the volume:

$ asmcmd volcreate -G DATA -s 100M testvol

View the volume information:

$ asmcmd volinfo –G DATA testvol

Make a mount point directory:

$ mkdir /u01/app/oracle/acfsdata/testvol

Make the file system (as root):

# mkfs -t acfs /dev/asm/testvol-403

Mount the file system to the mount point:

# mount -t acfs /dev/asm/testvol-403 /u01/app/oracle/acfsdata/testvol


Creating an ASM Dynamic Volume with Enterprise Manager

Managing Dynamic Volumes with SQL*PLUS

SQL> ALTER DISKGROUP DGROUPA ADD VOLUME asmvol1 SIZE 10g;

SQL> ALTER DISKGROUP DGROUPA RESIZE VOLUME asmvol1 SIZE 15G;

SQL> ALTER DISKGROUP DGROUPA DROP VOLUME asmvol1;

SQL> ALTER DISKGROUP DGROUPA ENABLE VOLUME asmvol1;

SQL> ALTER DISKGROUP ALL DISABLE VOLUME ALL;

SQL> ALTER DISKGROUP DGROUPA MODIFY VOLUME asmvol1 USAGE 'acfs';

Registering an ACFS Volume

As the grid owner, register the volume:

$ /sbin/acfsutil registry -a -f /dev/asm/volume1-167 /u01/app/oracle/acfsdata/volume1


Creating an ACFS Volume with ASMCA 

 

Extending ASMCMD for Dynamic Volumes

Create an ACFS file system:

# mkfs -t acfs /dev/asm/vol1-nnn

Mount an ACFS file system:

# mount -t acfs /dev/asm/vol1-nnn /oracle/cluster1/myacfs

Unmount an ACFS file system:

# umount /oracle/cluster1/myacfs

Check and repair an ACFS file system:

# fsck -t acfs /dev/asm/vol1-nnn

ACFS Platform-Independent Commands

acfsutil info	Display new ACFS file and file system features (ACFS, snapshots, registry, size) and information.
acfsutil snap	Create and delete ACFS snapshots.
acfsutil registry	Register an ACFS file system with the ACFS mount registry (that is, cluster-wide fstab).
acfsutil rmfs	Remove an ACFS file system.
acfsutil size	Resize an ACFS file system.
acfsutil tune	View or modify ACFS tune-ables.

Striping Inside the Volume

Inside a volume, space is allocated based on:

• Stripe column – the number of stripes (default 4)
• Stripe width – the size of each stripe (default 128 KB)

Stripe column and stripe width are properties that are set when you create the volume.

The stripe column is used both for allocating the initial space for the volume but also for the way the space inside the volume is used.

The volume will be formatted for a file system. As the file system allocates space for files, the files will be stored in pieces the size of stripe width.

If defaults are used, they are stored in 128 KB pieces. Each piece goes into the next volume extent. Because the volume extents are placed on disks in the disk group in the same manner, as the pieces are placed in the extents, the pieces are spread across the disks in the disk group.

ADVM Processes

A small number of processes will be added to the ASM instance when a volume is enabled. These processes are not started when there are no volumes configured.

• The Volume Driver Background (VDBG) process forwards ASM requests to lock or unlock an extent for rebalancing, resize the volume, offline the disk, add or drop a disk, and force and dismount a disk group to the dynamic volume manager driver. The VDBG is a fatal background process so the termination of this process brings down the ASM instance.
• Volume Background (VBGn) processes wait for requests from the dynamic volume manager driver that need to be coordinated with the ASM instance. An example of such a request would be opening or closing an ASM volume file when the dynamic volume manager driver receives an open for a volume (possibly due to a file system mount request) or close for an open volume (possibly due to a file system unmount request). The unplanned death of any of these processes does not have an effect on the ASM instance.
• Volume Membership Background (VMB) coordinates cluster membership with the ASM instance.

ASM Cluster File System

The ASM cluster file system (ACFS) extends Automatic Storage Management (ASM) by providing a robust, modern, general-purpose, extent-based, and journaling file system.

ACFS provides support for files such as application binaries, report files, trace files, alert logs, and other application data files. With the addition of the ACFS, ASM becomes a complete storage management solution for both Oracle database and non-database files.

ACFS scales from small files to very large files (Exabyte) and supports a large number of nodes in a cluster. ACFS is an extent-based file system to provide high performance and utilizes a log-based metadata transaction engine for file system integrity and fast recovery.

ACFS file systems can be exported to remote clients through industry standard protocols such as NFS and CIFS.

ACFS eliminates the need for third-party cluster file system solutions and simplifies management for all file types in single node and grid computing environments.

ACFS supports dynamic file system expansion and contraction without any down time. ACFS makes use of the ASM mirroring and striping features in addition to hardware RAID functionality. ACFS can be managed with tools familiar to the DBA: Enterpriser Manager, SQL*Plus, and ASMCMD.

ACFS is integrated with the native OS file management tools, so the file system is transparent to the users. ASMCMD has UNIX-like syntax and command names that should be familiar to the system administrator and storage administrator. A new graphical tool (asmca) has been introduced to manage ASM and ACFS.

ACFS and ADVM Architecture Overview

In 10g, Automatic Storage Management (ASM) is both a file system and volume manager built specifically for Oracle database files.

Oracle 11g Release 2 includes the ASM dynamic volume manager (ADVM) and Oracle ASM cluster file system (ACFS).

ADVM provides volume management services and a standard disk device driver interface to clients. Clients, such as file systems and other disk-based applications, issue I/O requests to ADVM volume devices as they would to other storage devices on a vendor operating system.

ADVM extends ASM by providing a disk driver interface to storage backed by an ASM file. The administrator can use the ADVM to create volumes that contain file systems. These file systems can be used to support files beyond Oracle database files such as executables, report files, trace files, alert logs, and other application data files.

With the addition of ADVM and ACFS, ASM becomes a complete storage solution of user data for both database and non-database file needs. ACFS is intended as a general file system accessible by the standard OS utilities. ACFS can be used in either a single server or a cluster environment.

Note: Oracle ACFS file systems cannot be used for an Oracle base directory or an Oracle grid infrastructure home that contains the software for Oracle Clusterware, ASM, Oracle ACFS, and Oracle ADVM components. Oracle ACFS file systems cannot be used for an OS root directory or boot directory.

ASM volumes serve as containers for storage presented as a block device accessed through ADVM. File systems or user processes can do I/O on this “ASM volume device” just as they would on any other device. To accomplish this, ADVM is configured into the operating system.

A volume device is constructed from an ASM file. ASM file extents map the ASM volume file to logical blocks located on specific physical devices. Additional processes are started as part of the ASM instance and serve as intermediaries between the ASM instance and ADVM. To use the ADVM driver, an ASM instance must exist with at least one disk group mounted that can be used to contain an ASM volume file.

An ASM volume is an ASM file. It inherits the properties of the ASM disk group and behaves similar to any other ASM file. ASM volume storage is automatically rebalanced whenever a storage configuration change occurs. This reconfiguration can be performed while an ASM volume is in use. Because ASM uses direct I/O, ASM volumes offer performance equivalent to raw disks.

An OS device file is created automatically when an ASM volume is created using either asmcmd, SQL*Plus, ASMCA, or the Enterprise Manager graphical interfaces. On Linux, this device file is created in the /dev/asm directory. You can configure both disk group mount and volume-enable operations to occur automatically upon ASM instance startup. The volume device file names are unique clusterwide and persistent across all nodes in the cluster that have an ASM instance running with the disk group mounted and volumes enabled.

Upon Linux system startup, the Oracle clusterware startup will load the drivers (oraclesacfs, oracleoks, and oracleadvm). The ASM instance is started by the ASM cluster registry service (CRS) agent, which will also mount the appropriate ASM disk groups and enable volumes. The CRS agent will then mount any ACFS file systems in the Oracle Cluster Registry (OCR).

Similar actions are performed on Windows.

ACFS file systems are accessed through OS file system tools and APIs on UNIX and Linux systems, and accessed through Windows file system tools and APIs on Windows systems. Remote access is supported using standard NAS file access protocols such as network file systems (NFS) and common Internet file system (CIFS) in support of heterogeneous file data sharing.

The ACFS File System and ADVM components are installed onto each host along with the other ASM components into the Grid Infrastructure home location. The ACFS components consist of drivers that are dynamically loadable OS modules, several command-line tools, and a set of processes that execute within the ASM instance. However, loading the ACFS drivers requires root privileges on UNIX/Linux and Administrator privileges on Windows. So, the configuration and loading of the ACFS drivers is performed by the root scripts associated with the Oracle Grid Infrastructure installation.

ACFS file systems are generally mounted on all cluster synchronization service (CSS) cluster members. In the event of a member failure, another cluster member will recover any outstanding metadata transactions on behalf of the failed member. In addition, any lock tokens held by the failed cluster members will be recovered and the failed member will be I/O fenced from the active CSS cluster. Following recovery, access by other active cluster members and any remote client systems may resume.