Analyzing communication latency using the Nectar communication processor

Abstract

For multicomputer applications, the most important performance parameters of a network is the latency for short messages. In this paper we present an analysis of communication latency using measurement of the Nectar system. Nectar is a high-performance multicomputer built around a high-bandwidth crosspoint network.Nodes are connected to the Nectar network using network coprocessor that are primarily responsible the protocol processing, but that can also execute application code. This architecture allows us to analyze message latency both between workstations with an outboard protocol engine and between lightweight nodes with a minimal runtime system and a fast, simple network interface (the coprocessor). We study how much context switching, buffer management and protocol processing contribute to the communication latency and we discuss how the latency is influenced by the protocol implementation. networks have to cover much larger distances and introduce errors in the data stream with non-zero probability. The communication protocols that recover from these errors introduce overhead, thus adding to the latency. This overhead however is, or should be, of the order of 10s of microseconds [7], and does not account for the order of magnitude difference in latency. In this paper we identify other sources of overhead based on measurements collected on Nectar.The Nectar network [3,11] consists of a high-bandwidth crosspoint network (Figure 1-1). Host are connected to the network through Communication Acceleration Boards (CABS) that are responsible for protocol processing. A 26-node prototype system using 100Mbit/second links has been operational since 1989 and has been used to parallelize several engineering and scientific applications [12]. 'he host-host latency over Nectar is about 200 microseconds. One of the goals of the prototype was to experiment with different protocol implementations and make the communication coprocessor customizable by applications. For this reason, we built the CAB around a general-purpose CPU with a flexible runtime system [8]. The application-level CAB-CAB latency is about 150 microseconds. The Nectar architecture allows us to study the communication between different types of hosts: