Towards Dynamic Resource Management with MPI Sessions and PMIx

Abstract

Job management software on peta-and exascale supercomputers continues to provide static resource allocations, from a program's start until its end. Dynamic resource allocation and management is a research direction that has the potential to improve the efficiency of HPC systems and applications by dynamically adapting the resources of an application during its runtime. Resources can be adapted based on past, current or even future system conditions and matching optimization targets. However, the implementation of dynamic resource management is challenging as it requires support across many layers of the software stack, including the programming model.In this paper, we focus on the latter and present our approach to extend MPI Sessions to support dynamic resource allocations within MPI applications. While some forms of dynamicity already exist in MPI, it is currently limited by requiring global synchronization, being application or application-domain specific, or by suffering from limited support in current HPC system software stacks.We overcome these limitations with a simple, yet powerful abstraction: resources as process sets, and changes of resources as set operations leading to a graph-based perspective on resource changes. As the main contribution of this work, we provide an implementation of this approach based on MPI Sessions and PMIx. In addition, an illustration of its usage is provided, as well as a discussion about the required extensions of the PMIx standard. We report results based on a prototype implementation with Open MPI using a synthetic application, as well as a PDE solver benchmark on up to four nodes and a total of 112 cores. Overall, our results show the feasibility of our approach, which has only very moderate overheads. We see this first proof-of-concept as an important step towards resource adaptivity based on MPI Sessions.

Malleability in Modern HPC Systems: Current Experiences, Challenges, and Future Opportunities

“…Monitoring solutions, for example, could be used here to find the right timing for such operations. Applications supporting advanced forms of checkpoint/restart or dynamic load balancing [24], [25] are good examples, being massively parallel and of an iterative nature with prescribed points where resource changes can be accommodated. Also, certain kinds of applications, e.g., those based on task farming or parallel-in-time codes based on the Parareal algorithm [26], can benefit from malleability without significant code changes.…”

“…II-C), for example, the user-level code must outline the reconfiguration phase and carefully define datamanagement procedures enabling the remapping of the various operands. Nonetheless, more generic support could, e.g., be devised either inside PETSc [25] that controls the data or task distribution of applications or directly through MPI [24], with a set of extensions for dynamic resources and AMR.…”

“…3) Dynamic Processes with Process Sets (DPP): A recent approach [24], [59], [60] introduced dynamic MPI Process interface extensions and an implementation based on Open MPI [61]. This approach follows the principles further described in [60] allowing, e.g., adding/removing processes to/from the application at runtime.…”

“…Recently, a prototype was developed, which extends the PRRTE, OpenPMIx, and Open MPI implementations to support a dynamic MPI Sessions interface [24]. These extensions allow applications to request the addition of processes to and removal of processes from the application during runtime.…”

“…This requires it to be programmable in an easy way. Here, we envision hiding this complexity in three different ways: 1) Provide malleability support within commonly used parallel software and parallel libraries, which lowers the bar for using malleability (see Huber et al [24], which uses a dynamic resource extension for p4est) or makes malleability almost transparent to the application developers using these updated software packages. 2) We envision a standardized layer between the application and MPI that provides various functionalities to again lower the bar for utilizing malleability.…”