Strategies for Fabrics
PCI Express Fabrics Break Through I/O Limitations
Integration of PCIe 3.0 with VPX platform improves performance tenfold with no porting efforts.
VINCENT CHUFFART, KONTRON
Page 1 of 1
Matching CPU and I/O performance is an ongoing challenge for evolving industrial embedded designs. High-performance data processing applications require greater and greater processing capabilities that must not be bottlenecked with limited I/O performance. Sensor processing applications such as video compression, medical imaging, wireless communications, and test & measurement are examples of fast growing areas that require a next-generation design solution to effectively correspond I/O bandwidth with processing speed.
VPX is gaining ground in these industrial scenarios—ushering in a new era of rugged embedded computing, and allowing board computers to move away from decades of parallel bus architectures that seriously limited I/O performance. VPX connectors and backplanes can carry multi-Gigahertz signals and enable systems where the bandwidth is no longer shared between boards. Combined with Generation 3 Peripheral Component Interconnect Express (PCIe 3.0), the result is a new breed of unparalleled applications for high-performance data processing platforms.
Redefining Embedded Performance
Initially designed to replace the older PCI and PCI-X, PCIe is a computer expansion card standard used throughout the computing and embedded devices industries. PCIe provides a high-speed, high-performance, point-to-point link for interconnecting devices, a design essential for data-hungry industrial applications and an alternative to the system-wide shared parallel bus architecture. Now in Gen3, PCIe technology has doubled the throughput per lane from Gen2’s 4 Gbits/s to 8 Gbits/s. Although the effective raw bit rate has only increased from 5 GHz to 8 GHz, by optimizing the efficiency of the raw encoding from 8b/10b to 128b/130b, the bandwidth is actually doubled. In addition, PCIe 3.0 maintains backward-compatibility with previous generations.
Developers of PC interconnects, graphics adapters, chip-level communications and others are enabled with even greater performance capabilities, matching CPU and I/O performance ratios. Consider all the data flowing within an M2M application on a diverse range of devices; PCIe 3.0 receives and transmits data on separate sets of signals, relieving data bottlenecks created by attempts to process an increasing amount of data in real time. Industrial applications such as medical, infotainment, digital signage and M2M/HMI are poised to benefit greatly from the combination of the VPX platform and PCIe 3.0, due to performance advances made possible by the resulting tremendous burst of I/O bandwidth and processing power.
Furthermore, thanks to broad IT market adoption of the technology, PCIe 3.0 is emerging as a leading serial link technology for VPX. High Performance Embedded Computing (HPEC), a typically mil/aero arena that is the first domain to benefit from this architecture, is realizing a tenfold increase in I/O bandwidth between computing boards offered by VPX backplanes. This holds significance for industrial embedded designers, and addresses an I/O performance gap so dramatic that only a fraction of current HPEC market applications really see significant application benefits in terms of accuracy, signal to noise improvement, cost and compactness of embedded computers.
PCIe 3.0 allows scalable, simultaneous, bi-directional transfers using 1 to 32 lanes of differential-pair interconnects. By grouping lanes, this standard can achieve high transfer rates similar to graphics adapters. Up to 32 Gbyte/s of bi-directional bandwidth on a x16 connector can be obtained with PCIe 3.0. It also enables low-overhead, low-latency data transfers. With both host-directed and peer-to-peer transfers, emulation of network environments can send data between two points without host-chip routing. These features make PCIe 3.0 an ideal solution not just to link high-bandwidth I/Os to a processor unit, but also to become a native communication link between computing devices in a multiprocessor environment, such as reconstructing 3D images from high-definition image sensors.
PCIe 3.0 does however require a specific skill set in order to offer the complete temperature range required in industrial applications. But, once mastered, the same bandwidth can be offered with fewer PCIe lanes—in turn accommodating small backplanes (3U) for smaller computers with the same performance. For example, x16 first-generation PCIe offers the same bandwidth as x4 PCIe 3.0, but requires four times the lanes on a backplane.
Protecting Embedded Software Investments
An existing distributed application might exchange information on gigabit Ethernet, implemented on most VME or CompactPCI platforms today. Alternatively, VPX platforms can implement TCP/IP protocol over the PCIe 3.0 infrastructure. The VPX system would incorporate VXFabric, Kontron’s open infrastructure that implements efficient inter-board communication at hardware speed, to tap into high-speed PCIe 3.0 bandwidth for data transfers simply by selecting a different IP address to connect to the other boards. No change is needed in software coding. This combination of technologies insulates applications from the complex, low-level details of the current generation of PCIe silicon management, and further prevents software from obsolescence.
VXFabric enables the use of standard communication protocols such as TCP/IP or UDP/IP based on its socket API. The API provides a thin layer of software that allows faster application development for IP-based transport over PCIe 3.0. From a hardware point of view, the architecture is based on several CPU boards, each featuring several processing cores, interconnected through PCIe via the VPX backplane, and using a PCIe 3.0 switch. Through VXFabric, any industrial application based on TCP/IP will run unmodified on this platform.
PCIe 3.0 switches offer the ability to combine different data types in a single converged pathway. Data (compute, communication, or storage) is created and consumed as PCIe on each of the slots in the rack, delivering efficiency both in hardware architectural and software usage. For example, built around PLX ExpressLane PCIe 3.0 switches, Kontron’s VX3042 and VX3044 Intel Core i7-based single board computers (SBCs) routinely achieve 5.6 Gbytes/s in data throughput between any boards in a VPX rack (Figure 1).
From a software perspective, VXFabric offers the equivalent of an Ethernet network infrastructure, including the IP socket programmatic interface, implementing layers that allow direct access of classic protocols such as TCP or UDP. Development and migration efforts are streamlined since the API requires no modification of existing applications. The end-user application does not even know it is using VXFabric or PCIe, but instead sees its usual TCP/IP sockets, just like a common Internet- or cloud-based application. This reduces development efforts and simplifies migration to VPX for industrial applications evolving to greater image or video processing performance, sensor data processing, or more rugged deployments such as outdoor digital signage or M2M implementations (Figure 2).
Kontron’s VXFabric allows existing applications written for TCP/IP sockets to use PCI Express for higher bandwidth communication. No code change required. The VXFabric code behaves for the system as an Ethernet device. Physically, on each Kontron board there is a PLX non-transparent bridge chip. A narrow collaboration between PLX and Kontron was necessary to use the PLX silicon features at their maximum. And each Kontron VPX board implements this part and can offer top TCP/IP performance on the backplane.
Due to the PCI fast link’s plug and play capability, the switch fabric moves data at an ultra-high speed. This solution enables deployment of high-performance—up to 6U OpenVPX—solutions, with better than 10 Gbit/s board-to-board connectivity. At the same time it facilitates integration of next-generation processor architectures. Further, PCIe’s performance as a native data bus in all modern processor chipsets delivers a key advantage—a broad PCIe-based software ecosystem with well-developed support for peripheral interconnects.
The Value of Switch Fabric
PCIe 3.0 in VPX systems relies on point-to-point connections between boards to manage high-bandwidth traffic. These connections require backplane routing specific to each application, in order to create connections between boards matching the target application data flows. The use of a switch-based fabric approach allows designers to seamlessly implement all routes to and from boards dynamically—in turn offering ample bandwidth to any application’s data flow, all with the same hardware.
Using this design approach, OEMs and their customers optimize total cost of ownership and further maintain a direct migration path forward from existing applications deployed today. Switch fabric enables a cost-effective bridge between current Gigabit Ethernet on the backplane (in industrial platforms such as VME, cPCI, VPX) and the next data plane generation of 10G and 40G Ethernet. Industrial applications seeking a performance jump can today access 10G and 40G performance in compact VPX-based systems featuring low power consumption and harsh environment capabilities; these same systems address all fast and low latency peer-to peer inter-computer node communication within a chassis.
This is a sea change for industrial computing, taking next-generation applications well beyond Gigabit Ethernet capabilities. The I/O performance advantages from VPX and PCIe 3.0 equate to exchanging the data of one DVD per second between boards. This promises to enable significant advances in size and resolution of images, for example, leading to improvements in imaging applications in manufacturing or medical environments. The rugged small size of these systems also means these features can be more readily used in mobile environments.
During the last 15 years, only Gigabit Ethernet evolved to enhance intra-communication bandwidth on backplanes. The 70MByte/s exchanges on VME backplanes were replaced by 120 Mbyte/s exchanges; this is the VITA 31 standard, also on the backplane, or with cables on the front. Alternative technologies such as InfiniBand or RapidIO were made available to offer more bandwidth. However, these more niche market technologies will not survive as long as mainstream technologies such as TCP/IP and PCIe. This is due partly to their development cost as well as their inability to offer Ethernet’s guarantee of a long performance lifetime. In turn, board computing systems had to rely on several different hardware and software solutions for high-speed serial link point-to point connections between boards. Due to the lack of market traction, none of them is expected to survive the next decade.
In contrast, PCIe is available in all chipsets such as Intel CPUs and bridges. There is no extra cost in terms of budget or power to use this efficient link versus other technologies that require extra silicon. PCIe native is also here to stay, visible in all modern computer architectures from first-generation PCIe in tablets or smartphones, to PCIe 3.0’s wide data path applications in large servers. Convergence is happening, and the same PCIe lanes going out of the board can be used to attach high-speed endpoints such as GPUs or FPGAs, which are already based on PCIe interfaces.
PCIe 3.0 and VPX Meet Long-Term Embedded Design Requirements
As VPX hardware guidelines are set for up to the next 20 years, OEMs and developers must select ideal communications protocols based on proven VPX standards such as VITA 46 and OpenVPX VITA 64. PCI Express, Gigabit Ethernet, Serial Rapid IO and many others can be used for intra-system communications, yet the challenge for OEMs is to choose an easy-to-use, yet fast and low latency communication protocol. PCIe 3.0 has emerged as a highly sensible backplane interconnect solution—enabling a new realm of image-intensive, small form factor applications.
Multi-gigahertz signals warrant systems where the full data plane bandwidth is no longer shared between boards. With VPX connectors and backplanes, each board is capable of one or more dedicated 10 Gigabit connections via Ethernet or PCIe. The rugged VPX platform enables PCIe 3.0’s high-speed connections in harsh environments, and VXFabric bridges the gap between this disruptive technology and currently deployed applications exchanging data on Gigabit Ethernet. Applications have top performance in a small envelope and avoid proprietary technologies. TCP/IP applications run unmodified on proven, ruggedized VPX platforms, protecting software investments over long-term deployments.
Industrial designers are taking their cue from HPEC applications, with developers focusing on the end application and relying on proven VPX and PCIe 3.0 to create a tenfold increase in I/O bandwidth in a small 3U VPX platform. High-definition medical imaging, video and display interfaces in digital signage, or complex sensor processing from M2M deployments all stand to benefit as well, with faster development, increased mobility and improved overall rugged performance.