TECHNOLOGY IN SYSTEMS
Managing the Internet of Things
Applying ATCA Hardware Platform Management to IoT Backend Systems
Hardware platform management is essential to the often Cloud-based network systems that enable the Internet of Things. The well-defined and designed ATCA management framework can be adopted by other form factors for management layer compatibility and savings across a number of different system architectures.
BY MARK OVERGAARD, PIGEON POINT SYSTEMS
Page 1 of 1
Discussions of the Internet of Things (IoT) typically focus on the “things”—that is, the millions or billions of small Internet-connected devices that constitute the “front-end” user interaction points in an IoT-oriented system. Just as crucial, however, is the back-end part of IoT, where potentially massive communication throughput and computation power is needed to service the needs of those millions or billions of “things,” possibly with stringent demands on the reliable availability of those services.
In their first decade, AdvancedTCA (ATCA) platforms have delivered high bandwidth, high function, high availability services around the world, and such platforms are certainly a candidate for the IoT back-end systems. For IoT systems where utility-class service availability is mandatory, serious management infrastructure is certainly required, and ATCA’s hardware platform management facility definitely qualifies as a foundation layer.
However, not all designers of such systems are ready to adopt ATCA wholesale. Here we introduce another option for those designers: adopt the management framework of ATCA, but make independent choices on physical form factors and other product aspects as necessary to fit application needs. This approach allows designers to leverage the successful worldwide usage of ATCA management architectures and subsystems in telecommunications and other industries, while retaining implementation freedom in other system aspects.
What is the conceptual management architecture on which ATCA is based?
Figure 1 shows the high level management architecture defined by the ATCA specification. The building blocks of the architecture include hot-swappable boards, with the potential for an optional rear transition module (RTM) connecting to each board to simplify I/O connections on the back of a shelf or chassis, plus other field replaceable units (FRUs), such as fan trays or power entry modules providing auxiliary services for a shelf. The complementary AdvancedMC specification defines hot-swappable modules that can optionally be hosted by the ATCA boards. The ATCA/AMC architecture also provides for various types of management controllers for the different types of building blocks in a shelf.
Figure 1: Building blocks and corresponding management controllers for an example shelf with ATCA/AMC-based management.
As shown in that figure, each shelf is supervised by a shelf manager (optionally with redundant instances), with a shelf management controller (ShMC) at its core. The shelf manager monitors the operation of managed elements in the shelf and represents the entire shelf to higher layers of management. Those higher layers are represented by a logical System Manager, which also might be responsible for managing many other shelves in a multi-shelf system.
Each board is monitored and managed at the low level by an intelligent platform management controller (IPMC), which also represents the board to the shelf manager over the I2C-based, dual redundant Intelligent Platform Management Bus called IPMB-0. Optionally, IPMCs can connect with the shelf manager via an Ethernet fabric in the shelf as well. Other FRUs, such as fan trays, can also have IPMCs.
Finally, boards can optionally host management-enabled, hot-swappable modules, each of which is monitored by a module management controller (MMC). The MMC represents the module to a carrier IPMC, which is like an IPMC, but with additional functionality to handle management-enabled subsidiary modules. Each MMC communicates with its supervising carrier IPMC via a local IPMB-L (a single I2C bus) and can optionally link to an in-shelf Ethernet as well. RTMs can be intelligent (monitored and represented by MMCs) or non-intelligent, with both variants also shown in the block diagram.
All of the architectural elements shown in the diagram and discussed above are independent of the physical form factors, such as size and the backplane (or module carrier board) interface implemented in the boards. They can even be independent to a large extent from the number of such boards in a shelf and not be limited to the number of boards allowed by ATCA. An ATCA-based management system that supports all these elements can be applied to a physical system framework that looks very different from ATCA.
Note, however, that a designer of one of these systems could choose to include a subset of board slots that are fully ATCA compliant. The billion dollar ATCA ecosystem provides numerous generally useful board offerings, such as cost-effective, high-performance, x86 CPU boards. Such boards may well be a welcome option, even if the bulk of a proprietary system is only ATCA-oriented at the management level. One benefit of such hybrid architectures is that the ATCA-compliant elements fit smoothly into them.
What functionality does an ATCA-based board level management controller provide?
Figure 2 illustrates the kind of services that an ATCA-based IPMC provides. The figure shows the outline of an ATCA board and an optionally rear transition module (RTM). An IPMC with equivalent services can be installed on essentially any reasonably sized board. Also in the figure is a photo of a reference IPMC implementation, represented to scale with an ATCA Front Board (8U high by 280 mm deep).
Figure 2: Representative functionality for an ATCA-based IPMC, shown in the context of an ATCA board.
As the figure shows, an IPMC can monitor and control hardware platform management attributes such as temperatures, power inputs, onboard power rails and front panel LEDs. There is also support for a handle switch (used during hot insertion/extraction of the board from a live system if that is needed). In addition, there is an interface to the “payload,” typically the main processor(s) on the board, such as a serious x86 processor. If relevant to the proprietary platform, there is support for an RTM managed by the main board.
There is also support for ATCA E-Keying, which enables an architecture where boards implementing different communication fabrics can automatically learn whether the boards they connect with via the backplane are protocol compatible. Many proprietary systems do not need this facility, since they’re not intended to support the wide range of communication fabrics that ATCA does.
Finally, there are provisions in ATCA IPMCs for them to connect with a network controller or switch to complement the default IPMB-0 that the shelf manager uses to communicate with the IPMCs in a shelf. There is an ecosystem of open specifications, both PICMG specifications and one from the Distributed Management Task Force (DMTF), which provide for management traffic with the IPMC to share the use of one or more Ethernet controllers that primarily support payload communication. Such a “LAN-attached” IPMC can support more and higher performance services over Ethernet than an IPMC that communicates solely over IPMB-0.Standardized LAN-Attached Management Controllers Yield xTCA Performance and Serviceability Gains, in the October 2012 issue of RTC, provides more background on these facilities.
The example reference IPMC shown here is from the Pigeon Point Board Management Reference (BMR) family, and comes with schematics and firmware source code, enabling easy adaptation to the needs of either ATCA or custom boards with ATCA-based management.
What facilities does a typical ATCA-based shelf manager include?
As mentioned above, an ATCA shelf manager is responsible both for monitoring the operation of the shelf and for providing an interface to the shelf for higher level management layers (what ATCA refers to as the logical System Manager), often via dual redundant Ethernet hubs or switches. Figure 3 shows these northbound shelf manager interfaces and the southbound interfaces inside the shelf.
Figure 3: ATCA-based shelf manager interfaces up to higher levels of management (often outside the shelf) and down to management controllers inside the shelf.
As shown, a typical shelf manager offers several northbound interface options. The Remote Management Control Protocol is part of the Intelligent Platform Management Interface (IPMI), an open specification that is widely used for management in the PC and server industries. (IPMI is the primary foundation layer for all the ATCA management controllers.)The Simple Network Management Protocol (SNMP) is also widely used for remote management and monitoring. Typically, there will be a Web interface (based on HTTP) and some sort of command line interface (CLI) as well. Finally, there can be a hardware platform interface (HPI), which is covered in the next section. In a proprietary application of ATCA-based management, the most suitable/applicable of these interfaces can be used to interface with the System Manager and/or human operators, with the others ignored, if appropriate.
A shelf manager can also support redundant instances (active and standby); a software and hardware redundancy interface (SRI and HRI) can be used to keep the two instances synchronized. Leveraging such synchronization, the standby instance can take over responsibility for the shelf if the active instance goes down for any reason. Such a switchover can even be invisible to System Manager clients, with key Internet Protocol (IP) addresses transferred from the active to the standby during the switchover.
Some key subsystems of a shelf manager are also shown in the diagram, such as those that handle cooling, track events and manage the FRUs in the shelf. A shelf adaptation layer allows for customization to the specifics of a given shelf and supports interfaces to the key functional blocks in the shelf, such as fans and power entry, as well as the IPMC-represented boards.
The shelf adaptation layer is especially important when an ATCA-based shelf manager, such as the widely used Pigeon Point shelf manager, is integrated into a non-ATCA shelf. Using I2C for “Behind the Scenes” Management, in the June 2009 issue of RTC, covers a variety of shelf adaptation layer strategies.
Can a System Manager handle both ATCA-based and other platforms?
A company that chooses to use ATCA management in a proprietary system may have other system types that are not based on ATCA, and may even have fully compliant ATCA systems as well. A range of system types like this may need to be supported concurrently, or over time, through successive generations of a major back-end system for the IoT.
One way to achieve System Manager commonality, and management unification overall, across such a range is to use the HPI as the lowest layer of the System Manager application(s). HPI was developed by the Service Availability Forum (www.saforum.org) for exactly this need. Figure 4 shows the concept.
Figure 4: A software platform (such as a System Manager) can use HPI to compatibly manage a wide range of platform types.
There is an open source implementation of HPI called OpenHPI (www.openhpi.org). It supports a plug-in architecture that allows interfacing to arbitrary hardware platforms, while the OpenHPI server that hosts the plug-in(s) offers a common management abstraction to the HPI client(s). An ATCA-based shelf manager with a built-in HPI server can support the OpenHPI client library interface. This architecture allows amortizing the System Manager investment (which can be large in a sophisticated high availability system such as might be used in an IoT back-end) across multiple platform types across projects or through time.
Pigeon Point IntegralHPI is an optional subsystem of the Pigeon Point shelf manager that supports the architecture shown in the figure. The architecture has been implemented by a Tier 1 Telecom Equipment Manufacturer (TEM) to share System Manager applications across multiple generations of ATCA and pre-ATCA hardware platforms.
The overall approach to ATCA-based management for proprietary systems described here is already being applied for major systems targeting Cloud-oriented computing, which is a natural operating context for IoT back-end systems.
Pigeon Point Systems