RTC Magazine

As the capabilities of computer software and hardware systems increase, along with the importance of the applications they support, the world becomes more and more dependent on their availability and reliability. Achieving the level of availability and reliability required of enterprise-class applications can be a substantial effort and expensive in development costs and the skills required, as well as having an impact on time-to-market.

A new class of infrastructure software is now available that has transformed high availability (HA) from a complex custom-programming task to a robust, easy-to-use, quick-to-deploy solution. This software transparently renders applications highly available and protects applications against process, node and network faults. No specialized expertise or knowledge in high-availability programming is required, and custom coding for recovery from faults is eliminated. And also most notably, this software can deliver the required “five nines” of uptime on lower cost commodity hardware platforms such as blade servers.

This new class of HA software can offer a flexible range of protection for applications based on checkpointing and restoration technologies. A choice of availability policies can be provided depending upon the criticality of the applications and the service required. A choice of checkpointing strategies is also provided through availability policies that can be modified in real time as desired.

For enterprise applications that must be highly available many features are essential including:

• Transparent and automatic checkpointing of single-threaded and multi-threaded application processes
• Infrastructure-provided and user-defined checkpointing
• Full and incremental checkpointing
• Infrastructure-initiated and application-initiated checkpointing
• Node and process fault detection
• Process deadlock detection
• Automatic restart of failed processes
• Automatic failover of failed nodes
• Support for redundant networks and failover to redundant network interface cards
• Subscription-based fault notification
• User-controlled migration of processes
• Automatic migration of processes to their home location after repair

HA infrastructure software provides automatic and transparent preservation and protection of both stateless and stateful processes, so that the users themselves need not be concerned with lack of high availability for their business- and mission-critical applications.

When discussing protection of enterprise-class applications and software infrastructure, certain basic terms and concepts are typically required.

A node is a computer, blade, rack-mount server or other hardware that runs a single copy of an operating system and one or more applications. A node can host primaries of one or more applications and backups of other applications.

A service unit constitutes one or more application processes running on the same node that collectively provide a service to the user. If one of the processes in a service unit is transferred to another node, all of the other processes in the service unit are also transferred to that node. The user specifies a list of nodes, referred to as the failover list, to which the processes in a service unit can be moved if a fault occurs.

Virtual IP addresses for the service units, also referred to as service addresses, are used to achieve location transparency. A service unit can have one or more service addresses. Service addresses are unique and are distinct from, but aliased to, the physical IP addresses of the node that hosts the service unit. When a service unit is migrated to another node, the service addresses of that service unit are moved to the other node, whereas the physical IP addresses remain with the original node. Users continue to interact with the migrated service unit through the service addresses.

Service units can have a home location, which is the node on which the service unit was originally started. If a service unit runs on a node other than its home node and the home node comes up, the service unit migrates back to its home node, which is referred to as failback.

Transparent High Availability

To be effective in today’s compute-intensive and clustered environments, the management software must render a wide variety of applications highly available and provide protection against process, node and network faults. It must separate the high-availability code from the application logic and allow an administrator to define policies for availability, checkpointing, fault detection and fault recovery. These policies should be easy to specify using, for example, XML and stored in a policy file, a copy of which is protected on each node thus avoiding any single point of failure.

The advantage of transparency is that it requires no specialized expertise or knowledge in high-availability programming, it eliminates the cost and complexity of custom coding for recovery from faults, and works in conjunction with standard operating systems (such as Linux) to make high availability economic and practical for a wide range of applications. For the enterprise, data center, telecommunications, defense/aerospace and medical industries, protection can be provided for both new and existing codes that must be highly available. This results in HA applications that are easier and quicker to deploy.

As an example, the Eternal Duration high-availability infrastructure comprises a library and an availability manager (AM) that reside between the operating system and the application, as shown in Figure 1. It also includes a system configurator (SC) and a fault notifier (FN), shown in Figure 2. The library is linked into each application process that must be highly available. The availability manager runs on each node in the system and is started automatically at boot time. The availability manager monitors the health of the local processes, manages the checkpoints of those processes, and restores a failed process locally or on a remote node. The system configurator allows the user to set the system policies and configuration information via the system configuration GUI. The fault notifier accepts fault reports from the availability managers and forwards them to the subscribers of fault reports.

In the compute environment the nodes are distinguished as either active or standby nodes. An active node is defined in the context of a particular service unit; it is the node on which the service unit is currently running. A node can be active for more than one service unit at a time. For a service unit that has a defined home location, its active node is normally its home location. If the active node for the service unit fails, the service unit is started on a designated standby node, which then becomes the active node for that service unit.

Efficient HA infrastructure software must support a variety of redundancy models. In the N+M redundancy model, there are N active and M standby nodes. A common special case is the N+1 redundancy model, where there is one standby node for all of the N active nodes. Another special case is the N+0 redundancy model, where there are no standby nodes. Figure 2 shows an example N+M configuration, where N=5 and M=2. In the NxM redundancy model, each of the N active nodes has M-1 dedicated standby nodes. A special common case is the 2xN redundancy model, where each active node has one dedicated standby node.

In the example above, the software provides high availability for an application by linking the library into each of the application processes and thus providing checkpointing, heartbeating and alarm services for that application. The library supports both infrastructure-provided and user-defined checkpointing. For infrastructure-provided checkpointing, it supports both full and incremental checkpointing. These checkpointing strategies determine what application state is checkpointed.

Each node in the system hosts an availability manager process, which is launched when the node starts up. When it comes up, the availability manager on a node reads the policy file, which is stored on the local disk of the node or in a shared file system. If one node is active and the other node serves as a standby for a service unit running on that node, the availability managers of the two nodes communicate with each other over the network..

The availability manager monitors the health of the processes in all of the service units on the node where it is running, using local fault detectors. The availability manager also manages the checkpoints of the processes in the service units running on its node. If a process fails, the availability manager restores the process locally, or restores all of the processes in the process’s service unit on the next standby node in the service unit’s failover list, depending on the settings in the policy file. The availability manager uses the most recent checkpoint to initialize the state of a process when it restores the process.

When a fault is detected, the availability manager sends a fault report to the fault notifier, if the fault notifier is installed in the system. The fault notifier notifies all registered subscribers and then sends an acknowledgment to the availability manager.

The availability manager accepts checkpoints from the application processes that are running on its local node. It stores the checkpoints locally to allow the processes in a service unit to be restored locally. It also transfers the checkpoints to the availability managers on the first available standby node on the failover list for that service unit. This allows those processes to be restored on that node, if a process in that service unit fails.

Example: A VoIP Application

Voice over IP (VoIP) is the transport of voice over the Internet Protocol (IP) and is the packet-switched alternative to the circuit-switched telephony of the past. VoIP must provide carrier-grade service, which means high availability (99.999% or better). In addition, it must provide scalability to hundreds of thousands of calls, predictable performance and high speech quality.

In this example, an open source H.323 Gatekeeper application is being protected by a high-availability infrastructure. The Opengate H.323 Gatekeeper is built on the OpenH323 H.323 Protocol Library Neither OpenH323 nor Opengate has high-availability features, such as checkpointing or redundancy, which makes Opengate an ideal case study. However, by using this new infrastructure software one can easily protect a stateful application that was never designed to be highly available. In a failure scenario, an application process is failed over to a standby application process, maintaining established VoIP call sessions.

H.323 is an ITU standard voice and video telephony architecture. In the simplest configuration, the gatekeeper converts telephone numbers to IP addresses. An endpoint (e.g., IP phone) must register its phone number and IP address with the gatekeeper before it can place or receive calls. After registering, an endpoint can initiate a call by requesting the gatekeeper to provide network access and address resolution for the called number. Once the gatekeeper has granted permission and resolved the phone number to an IP address, the endpoint can send call signaling directly to the other endpoint, or it can send call signaling via its gatekeeper. However, the gatekeeper determines which path the call signaling will take. In the case study, the system is configured to use direct endpoint-to-endpoint call signaling. The gatekeeper uses connectionless protocols based on UDP/IP, which the infrastructure software can restore automatically.

Figure 3 shows a time sequence diagram of a call setup performed in this manner. The originating endpoint first requests access permission from its gatekeeper. When the gatekeeper grants permission, and has performed address translation, the endpoint commences call signaling to the remote endpoint. The remote endpoint immediately returns a Call Proceeding message prior to asking its gatekeeper for permission to handle the call. By sending back the Call Proceeding message immediately, the endpoint signals to the call originator that it received the Setup message successfully. The called endpoint then requests permission from its gatekeeper to handle the call. The message exchange begins immediately after the Connect message.

If the gatekeeper process, or the node on which it is running, crashes, the endpoints will not be able to place calls. If the gatekeeper is restarted, an endpoint must re-register with the gatekeeper before calls can be placed to it. Most H.323 telephones periodically heartbeat the gatekeeper; however, it might take an unacceptably long time for all phones managed by a gatekeeper to re-register.

By checkpointing the state of the gatekeeper and automatically restoring the gatekeeper process from the checkpoint, either locally or on another node, service is immediately restored after the gatekeeper process or a node crashes without the need for all phones to re-register. Established calls are preserved and once the gatekeeper process is restored, new calls can be established.

To reiterate, this newly realized high-availability service is achieved for an application not previously designed for high availability by simply registering it for protection. This case study illustrates the easy-to-learn and easy-to-deploy nature that makes such HA infrastructure software so powerful in real-life situations.

Eternal Systems
San Jose, CA.
(408) 369-7400.
[www.eternal-systems.com].