RTC Group, Inc.

A High-Availability Ecosystem from the Service Availability Forum

By: Bill Swortwood, Motorola, Steve Dake, MontaVista Software, Adam Schmidt, Force Computers and Timo Jokiaho, Nokia–for the Service Availability Forum

A complementary specification is being added to the existing AIS and HPI specifications, which acts as an umbrella to tie them together in the fields of configuration, management and event notification in order to meet high-availability requirements.

The Service Availability Forum (SA Forum) mission in the communications and computing marketplace is to foster an ecosystem that enables the use of commercial off-the-shelf building blocks in the creation of high-availability network infrastructure products, systems and services. To achieve this, the SA Forum develops and publishes high-availability and management software interface specifications, and promotes and facilitates their adoption by the industry.

The SA Forum specifications include three levels. The Application Interface Specification (AIS) defines the interface between the applications and the high-availability (HA) middleware making each independent of the other, thus providing for a very robust HA stack management.The Hardware Platform Interface (HPI) defines the interface between the hardware and the HA middleware and makes each independent of the other. The Systems Management Specification (SMS) defines the interfaces to access the monitor and control aspects of the AIS and the HPI interfaces, as well as a comprehensive notification interface for HA systems. In addition to the AIS and HPI, the forthcoming Systems Management Specification will provide management and high availability. Figure 1 shows how the three specifications will work together.

Application Interface Specification

Applications today are often developed as one application that runs on a single node. In a typical system, this exposes four possible areas that could fail: the application, the middleware, the operating system and the hardware.

The SA Forum AIS improves service availability by taking advantage of redundant hardware and software components in a distributed system. The AIS defines a C language API for six services that together provide a distributed mechanism for supporting cluster membership, application failover, checkpointing, event distribution, messaging and distributed locking. The approach of the AIS to increase service availability is to mask the four possible failures by redundantly distributing the application and middleware across the same or multiple nodes.

Consider an application that runs on two nodes. If any of the four possible failures occurs on one of the nodes, the remaining node survives to provide service to the user of the application. Providing more than one standby node improves service availability dramatically. For applications to failover, they must make use of the Availability Management Framework (AMF). The AMF specifies a distributed system model and policy to describe the actions taken when a failure is detected or reported.

The system model of the AIS is provided by a container called a service group. A service group is a collection of service units. A service unit is a collection of one or more software components, or processes combined to deliver a service. Service units are assigned a component service instance (CSI), which is either active or standby. In Figure 2, the active service unit X contains components A and B. The standby service unit Y contains components C and D. If the active service unit X failed, service unit Y would become active.

 

A number of alternative policies are provided to specify which service units are activated during failover. The AIS explains these policies in great detail. One of these policies, for example is the N+M policy, which specifies that N service units will be active, and M will be standby. If an active CSI fails, a standby CSI will be assigned the active state. There are several mechanisms that will direct a standby CSI to enter the active state. Failure to respond to a health check will activate the standby unit. In addition, the crash of a component or an error report via the API, as well as the unregistration of a component via the API will bring the standby unit online.

With the AMF, stateful application failover is made possible with the addition of checkpointing. Checkpointing is the action of saving the active component’s state into a checkpoint section. When a standby service unit is activated, it can read the standby state and start at the last known good state of the active application. The Checkpoint API provides functions that create, delete, read or write a checkpoint. It also allows the automatic cleanup of checkpoints and the iteration of checkpoint sections.

Through the combination of the availability management framework API and the checkpoint API, redundant software can be developed that fully recovers component state during failover. There are other services that are necessary to provide a true distributed system. Since multiple components can exist in one service unit, it is useful for them to communicate. There are two services for communication. The first service is the Event API that provides:

The second service is the Message API that provides:

An API is provided to synchronize between components in a CSI. The Distributed Lock API provides:

The final service provides a view of the cluster nodes. The Cluster Membership API provides:

The AIS provides these services to develop distributed applications that increase service availability by distributing redundant applications across redundant platforms.

Hardware Platform Interface

The SA Forum HPI specification provides an industry standard interface to monitor and control highly available communications and computer systems platforms (Figure 3). The newly released version of the specification, HPI B.01.01, improves the specification in various areas. It is based on feedback from adopters of the original HPI specification and adds recommended functionality and flexibility. The HPI provides two different representations of a managed system: a physical view and a management view.

The physical view uses an entity identifier to describe a physical item in the system, as well as a hierarchical “entity path”, which describes a containment of the components within each other. Examples of entity paths are “Power supply 3, Chassis 2, Rack 1” or “Blade 5, Chassis 3, Rack 2.” In the updated specification, the mapping of the entities to physical system components was enhanced to better support bladed servers and ATCA systems. The “entity schemas” for various equipment families will be replaced by more detailed descriptions in separate documents.

The management view was also enhanced in multiple areas. The earlier specification, HPI A.01.01, defined four types of management instruments: sensors, controls, entity inventory repositories and watchdog timers. HPI B.01.01 adds annunciators. Their function is to provide an abstract interface to the platform’s alarm hardware, such as specific LEDs on the front panel, or sirens. Annunciators can be used in auto-mode, where the platform automatically adds and deletes announcements; in user mode, allowing the HPI user to manage the alarm announcements; or in shared mode.

HPI B.01.01 simplifies the representation and the usage of sensors. Most notably, the notion of raw values was deleted, so that now all values that sensors return are meaningful and independent from the implementation. The usage of entity inventory repositories (which in HPI B.01.01 are called inventory data repositories) was also simplified. It is now possible to check if the inventory data was updated between subsequent reads.

The management instruments are contained in resources, which are contained in domains. A resource is a set of management instruments associated with one or more entities. For example, a management controller for a set of power supplies, fans, boards, etc. might be modeled as a resource that contains sensors and controls. A resource supporting hot swappability can also have a hot swap state machine assigned to it. The state machine was improved in HPI B.01.01. A detailed mapping to various systems will be provided later in separate documents.

Domains group the resources, providing a single access point to the HPI user for the management of a system, and for discovery of the available instruments. HPI B.01.01 clarifies the usage of domains, introducing a new domain reference table that describes the relationship between them. Simple domain architectures are possible, with one domain containing resources only, as well as peer architectures with two domains containing the same resources (modeling redundant domains), and a tiered model where one domain includes other domains. Overlapping of resources is generally only allowed in the peer architecture.

Another added feature to HPI B.01.01 is the management of alarms. Each domain maintains a domain alarm table (DAT), which contains entries for each active alarm. For instance, an alarm entry in the DAT is made if a sensor asserts a significant event state (like over-temperature), or if there is a resource failure.

Although it was necessary to change the C API itself, the changes are straightforward and simple to incorporate for vendors implementing the HPI on their platforms and for framework programmers using the API to access the platforms’ features. Developers will certainly find the improved interface easier to use and more consistent. The specification is stable and will serve as the baseline for future certification testing and other specifications released by the SA Forum.

Systems Management Specification

Recently the SA Forum embarked on enabling distributed management interfaces and defining a notification infrastructure for high-availability clusters. This work will manifest itself in the SMS next year. The overall goal of the SMS is to address the administrative operations and management of cluster configuration, platform system model configuration, deployment and statistical data monitoring.

The SMS will provide critical monitor and control access to AIS and HPI facilities for administration of a SA Forum-enabled cluster. This is accomplished with the leverage of two industry standard distributed management protocols. Simple Network Management Protocol (SNMP) will allow rapid integration of the SA Forum administrative distributed interfaces into commonly available network management applications. The Common Information Model (CIM) effort will allow web-based and cross-discipline policy management by leveraging emerging network, platform and networked storage standards. CIM provides a consistent information model that promotes interoperability between management vendors and a management framework for administrators.

The CIM Schema incorporates concepts from other standards. For example, the CIM hierarchy organizes around the Internet Engineering Task Force’s (IETF) MIBs and the International Telecommunication Union (ITU) standards. Existing standards are not ignored or dismissed; they are instead reused and mapped into CIM.

The definition of distributed interfaces with both of these technologies enables the use of the “best tool for the job.” This allows the implementer to make utilization decisions based on their current management application infrastructure and needs, while allowing for evolution of and melding to more advanced management tools as they emerge in the marketplace. These distributed interfaces are intended to expose the management and control aspects of the SA Forum platform, the AMF and the HPI.

The SA Forum is collaborating with the Distributed Management Task Force (DMTF) to realize the models represented by these specifications. In addition, SA Forum member companies have also assisted in accelerating HPI SNMP agents by collaborating with open source efforts. This has resulted in the realization of an early version of an HPI 1.0 SNMP agent. Going forward, the SA Forum intends to develop HPI 1.1-compliant SNMP interfaces.

A notification framework that allows for collection, logging, filtering and forwarding of events, alerts, alarms, traps and application notices is currently being defined. It is expected to incorporate aspects of the ITU-T X.733 specification, and allow for integration with IETF and DMTF defined event services. The goal for the notification framework is the consolidation of all cluster-provided and platform-derived event information. The viewing and filtering of current and historic logged events will be provided. The automatic notification service recovery, automated clearing of undesired event reports and internationalization of the notification system are also being considered as design objectives.

The SMS is a complementary specification that acts as an umbrella to tie together the already existing HPI and AIS specifications in the fields of configuration, management and event notification. The HPI and AIS specifications will have extensions to support the requirements coming from the SMS in these domains to provide a consistent management view to both hardware management and availability areas. The SMS will continue to evolve to support high-availability needs, and address the management structure of the HPI, AMF and AIS APIs.

The SA Forum is working on an architectural model to ensure that the specifications already published and any future specifications are consistent with each other, and the related implementations will have a good basis to work together. The architecture work identifies and prioritizes the interface standards and specifications required to ensure that they achieve useable service availability standards within a viable supplier ecosystem. This will be built upon portability and the interchangeability of hardware, middleware and application services

The Service Availability Forum.
[www.saforum.org/specification].

© 2009 RTC Group, Inc., 905 Calle Amanecer, Suite 250, San Clemente, CA 92673