Designing MPI Library with Dynamic Connected Transport (DCT) of InfiniBand: Early Experiences

Subramoni, Hari; Hamidouche, Khaled; Venkatesh, Akshey; Chakraborty, Sourav; Panda, Dhabaleswar K.

doi:10.1007/978-3-319-07518-1_18

Cited by 19 publications

(3 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In software, many MPI implementations allow for changing from a static to a dynamic connection setup. In hardware, new connection modes have been added, such as the Extended Reliable Connected (XRC) Transport Service that enables processes running on the same node to share certain connections [10] and the Dynamically Connected (DC) Transport service that hands over the connection management to the HCA, which then sets up or tears down connections dynamically as required [11]. However, if the number of connections is low, as in stencil-type applications, these new modes are not necessary and the traditional reliable connected (RC) mode still yields best performance.…”

Section: Connection-oriented Vs Connectionless Communicationmentioning

confidence: 99%

DD-αAMG on QPACE 3

Georg¹,

Richtmann²,

Wettig³

2018

EPJ Web Conf.

View full text Add to dashboard Cite

We describe our experience porting the Regensburg implementation of the DD-αAMG solver from QPACE 2 to QPACE 3. We first review how the code was ported from the first generation Intel Xeon Phi processor (Knights Corner) to its successor (Knights Landing). We then describe the modifications in the communication library necessitated by the switch from InfiniBand to Omni-Path. Finally, we present the performance of the code on a single processor as well as the scaling on many nodes, where in both cases the speedup factor is close to the theoretical expectations. IntroductionThe lattice QCD (LQCD) community has traditionally been an early adopter of new computing and network architectures. This typically requires major efforts porting simulation code or even communication libraries. The Regensburg lattice group (RQCD) has been involved in such efforts, as well as supercomputer development, for more than a decade. While the first computer in the QPACE series [1,2] was based on IBM's Cell processor and an FPGA-based custom interconnect, the subsequent machines are using Intel's Xeon Phi series with standard interconnects (see Sec. 2.1). To satisfy the increasing demands of the RQCD physics program we use a state-of-the-art method, DD-αAMG [3], to solve the discretized form of the Dirac equation. The high-performance implementation of this solver on QPACE 2 is described in [4][5][6][7]. The present contribution focuses on the software efforts we made to efficiently run this implementation on QPACE 3. This paper is structured as follows. In Sec. 2 we give an overview of QPACE 3 and highlight the differences to QPACE 2 in terms of processor and network. We discuss the network technology in some detail because it has changed rather drastically. In Sec. 3 we describe how our solver and our communication library were adapted to the new technologies. In Sec. 4 we present single-node and multi-node benchmarks of the solver on QPACE 3 and compare the results with numbers obtained on QPACE 2. In Sec. 5 we conclude and give an outlook on future work. QPACE 3 2.1 OverviewWhile QPACE 2 [8] is based on the Knights Corner (KNC) version of the Intel Xeon Phi processor series and an FDR InfiniBand network, its successor QPACE 3 utilizes the current Xeon Phi processor, Speaker,

show abstract

Section: Connection-oriented Vs Connectionless Communicationmentioning

confidence: 99%

DD-αAMG on QPACE 3

Georg¹,

Richtmann²,

Wettig³

2018

EPJ Web Conf.

View full text Add to dashboard Cite

show abstract

“…For instance, Subramoni et al in [9] study the benefits of using the new Dynamically Connected (DC) InfiniBand transport protocol, showing great improvements for both synthetic benchmarks and production applications. Another such example can be found in the work by Wang et al in [10], where it is proposed a tuned GPU to GPU communication design for InfiniBand clusters. Nevertheless, although these proposals improve performance, both of them are focused on tuning MPI libraries, whose requirements differ from general applications.…”

Section: Related Workmentioning

confidence: 99%

Tuning remote GPU virtualization for InfiniBand networks

Reaño

Silla

2016

J Supercomput

View full text Add to dashboard Cite

In the past few years, a tendency towards using InfiniBand networks to interconnect high performance computing clusters can be observed. Thus, most of the supercomputers appearing in the TOP500 list either use Ethernet or InfiniBand interconnects. Regarding the latter, the complexity of the InfiniBand programming API (i.e., InfiniBand Verbs) makes it difficult for applications to get the maximum performance of these networks. In this paper we expose how we have tuned a remote GPU virtualization framework whose communications module is implemented using InfiniBand Verbs. The net result is a noticeable increase in the performance of this framework, significantly reducing the gap between remote and local GPUs.

show abstract

“…For instance, Subramoni et al in [9] study the benefits of using the new Dynamically Connected (DC) InfiniBand transport protocol, showing great improvements for both synthetic benchmarks and production applications. Another such example can be found in the work by Wang et al in [10], where it is proposed an optimized GPU to GPU communication design for InfiniBand clusters. Nevertheless, although these proposals improve performance, both of them are focused on optimizing MPI libraries, whose requirements differ from general applications.…”

Section: Related Workmentioning

confidence: 99%

InfiniBand Verbs Optimizations for Remote GPU Virtualization

Reaño

Silla

2015

2015 IEEE International Conference on Cluster Computing

View full text Add to dashboard Cite

Abstract-The use of InfiniBand networks to interconnect high performance computing clusters has considerably increased during the last years. So much so that the majority of the supercomputers included in the TOP500 list either use Ethernet or InfiniBand interconnects. Regarding the latter, due to the complexity of the InfiniBand programming API (i.e., InfiniBand Verbs) and the lack of documentation, there are not enough recent available studies explaining how to optimize applications to get the maximum performance from this fabric. In this paper we expose two different optimizations to be used when developing applications using InfiniBand Verbs, each providing an average bandwidth improvement of 3.68% and 217.14%, respectively. In addition, we show that when combining both optimizations, the average bandwidth gain is 43.29%. This bandwidth increment is key for remote GPU virtualization frameworks. Actually, this noticeable gain translates into a reduction of up to 35% in execution time of applications using remote GPU virtualization frameworks.

show abstract

Designing MPI Library with Dynamic Connected Transport (DCT) of InfiniBand: Early Experiences

Cited by 19 publications

References 10 publications

DD-αAMG on QPACE 3

DD-αAMG on QPACE 3

Tuning remote GPU virtualization for InfiniBand networks

InfiniBand Verbs Optimizations for Remote GPU Virtualization

Contact Info

Product

Resources

About