Performance evaluation of the Sun Fire Link SMP clusters

Abstract

As symmetric multiprocessors become commonplace, the interconnection networks and the communication system software in clusters of multiprocessors become critical to achieving high performance. Recently, Sun Microsystems has introduced a new system area network, Sun Fire Link interconnect, for its Sun Fire cluster systems. Sun Fire Link is a memory-based interconnect, where Sun MPI uses the Remote Shared Memory (RSM) model for its user-level internode messaging protocol. In this paper, we present the overall architecture of the Sun Fire Link interconnect, explain how communication is done under the RSM, and describe the Sun MPI implementation. We provide an in-depth performance evaluation of the Sun Fire Link interconnect cluster of four Sun Fire 6800s at the RSM layer, MPI microbenchmark layer, and the application layer. Our results include the performance of the Remote Shared Memory API primitives, MPI overhead on top of the RSM, latency and bandwidth under different communication modes, parameters of the LogP model, collective communications, permutation communications, and two applications (Sphot and SMG2000) from the ASCI Purple suite under the MPI, OpenMP, and MPI-OpenMP programming paradigms. Our results indicate that put has a better performance than get on this interconnect. The Sun MPI implementation achieves an off-node latency of up to 5 microseconds. This is comparable to the Quadrics (QsNet II), and Myrinet (E-card), and better than the InfiniBand. The unidirectional and bidirectional bandwidths are 695 MB/s, and 660 MB/s, respectively. The QsNet II, and InfiniBand achieve higher bandwidths. The LogP parameters indicate the network interface is less capable of offloading the host CPU when the message size increases. The performance of our applications under MPI is better than the OpenMP version, and equal or slightly better than the mixedmode. In general, the Sun Fire Link cluster performs relatively well in most cases. I. INTRODUCTION Clusters of Symmetric Multiprocessors (SMP) have been regarded as viable scalable architectures to achieve supercomputing performance and to meet the ever-increasing demand for high-performance computing. There are two main components in such systems: the SMP node, and the communication subsystem including the interconnect, and the communication system software. Considerable work has gone into the design of SMP systems, and several vendors such as IBM, Sun, Compaq, SGI, and HP offer small to large scale shared memory systems. Sun Microsystems has introduced its Sun Fire systems in three categories of small, midsize, and large SMPs, supporting two to 106 processors, backed up with its Sun Fireplane interconnect [1] used inside the Sun UltraSPARC III Cu. The Sun Fireplane interconnect uses one to four levels of interconnect ASICs to provide better sharedmemory performance. All Sun Fire systems use point-to-point signals with a crossbar rather than a data bus. The interconnection network hardware and the communication system software are the keys to the performance of cl...

Optimising Up*/Down* Routing By Minimal Paths

“…Some commercial interconnects for NOWs are Myrinet [11], Servernet II [1], Autonet [10], Gigabit Ethernet [14], and InfiniBand [4]. And several high-performance interconnects have been recently introduced for NOWs, including the Quadrics QsNet [2], and QsNet II [12], and Sun Fire Link [16].…”

An evaluation of the Myrinet/GM2 two-port networks

“…This shows MPICH-GM has very well been ported to GM. A quick comparison with other high-performance interconnect are as follows: Quadrics QsNet: 4.6µs [18], and QsNet II: ~3µs [1], Myrinet D-card: 6.7µs [13], InfiniBand: 6.8µs [13], and Sun Fire Link: 5µs [19].…”

“…The results verify that the Myrinet with twoport outperforms the one-port. A quick comparison with other high-performance interconnect are as follows: Quadrics QsNet: 308MB/s [18], and QsNet II: ~900MB/s [1], Myrinet D-card: 471MB/s [13], InfiniBand: 841MB/s [13], and Sun Fire Link: 792MB/s [19]. The Myrinet aggregate bandwidth when simultaneous messages are in transit between pairs of send and receive processes is shown in Figure 9.…”

“…Research groups in academia and industry have been studying the performance of the interconnection networks at the user-level, the MPI level, and the application level [13,14,1,18,4,11,19]. In [13], the authors assessed the performance of the MPI implementation on InfiniBand, Myrinet, and Quadrics.…”

“…Several high-performance interconnects have been recently introduced including the Quadrics QsNet [18], and QsNet II [1], InfiniBand [16], Myrinet [5], SCI [8], and Sun Fire Link [19]. Each one of these interconnects provides different levels of performance, programmability, and integration with the operating systems.…”