Efficient Scalable Computing through Flexible Applications and Adaptive Workloads

Abstract

Abstract-In this paper we introduce a methodology for dynamic job reconfiguration driven by the programming model runtime in collaboration with the global resource manager. We improve the system throughput by exploiting malleability techniques (in terms of number of MPI ranks) through the reallocation of resources assigned to a job during its execution. In our proposal, the OmpSs runtime reconfigures the number of MPI ranks during the execution of an application in cooperation with the Slurm workload manager. In addition, we take advantage of OmpSs offload semantics to allow application developers deal with data redistribution. By combining these elements a job is able to expand itself in order to exploit idle nodes or be shrunk if other queued jobs could be initiated. This novel approach adapts the system workload in order to increase the throughput as well as make a smarter use of the underlying resources. Our experiments demonstrate that this approach can reduce the total execution time of a practical workload by more than 40% while reducing the amount of resources by 30%.

DMRlib: Easy-Coding and Efficient Resource Management for Job Malleability

“…The Dynamic Management of Resources (DMR) API [15] implements a communication layer between the OmpSs runtime (Nanos++) and Slurm that allows MPI applications to be resized. The DMR API relies on the off-load semantics of OmpSs [16] for automatically handling processes and data redistribution, and on Slurm for managing and reallocating resources.…”

“…As the works based on SCR extensions [5], EasyGrid AMS [10], ULFM [5], and ReSHAPE [12], do not publicly provide any example on how to implement malleability, we limit the study to PCM [4], AMPI [7], Flex-MPI [11], Elastic MPI [2] and DMR [15].…”

“…With those results, we obtain a relative factor of execution time decrease, referred to as gain difference. The choice of the gain difference is justified by its capacity to provide a fair balance between the number of resources allocated and the speedup obtained, as proven in several publications [15], [3], [9]. The value of this heuristic decreases rapidly as the number of required nodes increases if there is not remarkable performance gain in between configurations.…”

Dynamic reconfiguration of noniterative scientific applications: A case study with HPG aligner

“…The DMR API reconfiguration policy considers queued jobs and available resources in order to perform a purely throughput-aware scheduling. It was evaluated on the CG, Jacobi, and the N-body problem (Iserte et al, 2017).…”

“…The main goal of this work is to demonstrate that by leveraging the Dynamic Management of Resources Application Programming Interface (DMR API; Iserte et al, 2017), out-of-target applications can be considered for malleability. Unlike other malleability frameworks, the DMR API provides the coding flexibility and programming easiness that are required to reconfigure most of the current target and out-of-target applications.…”