Gigabit Nectar: Architecture and Performance.

Abstract

<£<?//<? O3 9Multicomputers consisting of off-the-shelf computers connected by commercial high-speed networks form an economically attractive computing platform for a large class of applications. However, while highspeed networks are fairly widely available (e.g. HIPPI and ATM), many computer systems have problems delivering this high bandwidth to the applications, thus limiting the class of applications that can be supported by multicomputers. The Gigabit Nectar project developed a network interface architecture that supports efficient high-bandwidth end-end communication. This architecture has been implemented for workstations (DEC Alpha) and distributed-memory systems (iWarp) and has been deployed in the Gigabit Nectar testbed. This report describes the Nectar network interface and its implementation and performance, and summarizes our application experience in the testbed.Recent advances in network technology have made it feasible to build high-speed networks using links operating at several 100s of Mbit/second. HIPPI networks based on the ANSI High-Performance Parallel Interface (HIPPI) protocol [1] are an example. HIPPI supports a data rate of 800 Mbit/second or 1.6 Gbit/second and almost all commercially available supercomputers have a HIPPI interface. As a result, HIPPI networks have become popular in supercomputing centers. In addition to HIPPI, there are a number of high-speed network standards in use, including ATM (Asynchronous Transfer Mode) [18] and Fibre Channel [2].As network speeds increase, it is important that host interface speeds increase proportionally, so that applications can benefit from the increased network performance. For bulk data transfer over high-speed networks, the sending and receiving hosts typically form the bottleneck, and it is important to minimize the communication overhead to achieve high application-level throughput. The communication cost can be broken up in per-packet and per-byte costs. The per-packet cost can be optimized [14,44,11], and for large packets, this overhead is amortized over a lot of data. However, the per-byte cost is not reduced by increasing the packet size. Moreover, the per-byte cost depends strongly on the memory bandwidth, which over time has not increased as quickly as CPU speed. As a result, it is mainly the per-byte costs that make high speed communication over networks expensive and that ultimately limit throughput as the network bandwidth increases.We have designed a host-network interface architecture optimized to achieve high application-to-application throughput for applications using the socket application programming interface (API) and the internet communication protocols. Our interface architecture is based on a Communication Accelerator Block (CAB) that provides support for key communication operations. The CAB is a network interface architecture that can be used for a wide range of hosts, as opposed to an implementation for a specific host. Two CAB implementations for HIPPI networks have been built by Network Systems Corporation (NSC...