MapReduce-Based RESTMD: Enabling Large-Scale Sampling Tasks with Distributed HPC Systems

Abstract

A novel implementation of Replica Exchange Statistical Temperature Molecular Dynamics (RESTMD), belonging to a generalized ensemble method and also known as parallel tempering, is presented. Our implementation employs the MapReduce (MR)-based iterative framework for launching RESTMD over high performance computing (HPC) clusters including our testbed system, Cyber-infrastructure for Reconfigurable Optical Networks (CRON) simulating a network-connected distributed system. Our main contribution is a new implementation of STMD plugged into the well-known CHARMM molecular dynamics package as well as the RESTMD implementation powered by the Hadoop that scales out in a cluster and across distributed systems effectively. To address challenges for the use of Hadoop MapReduce, we examined contributing factors on the performance of the proposed framework with various runtime analysis experiments with two biological systems that differ in size and over different types of HPC resources. Many advantages with the use of RESTMD suggest its effectiveness for enhanced sampling, one of grand challenges in a variety of areas of studies ranging from chemical systems to statistical inference. Lastly, with its support for scale-across capacity over distributed computing infrastructure (DCI) and the use of Hadoop for coarse-grained task-level parallelism, MapReduce-based RESTMD represents truly a good example of the next-generation of applications whose provision is increasingly becoming demanded by science gateway projects, in particular, backed by IaaS clouds.

Hadoop‐based replica exchange over heterogeneous distributed cyberinfrastructures

“…Indeed, a lower developmental cost and benefits of well‐established Hadoop and MR are attractive for many HPC applications of similar algorithmic characteristics. As we examined in the conference paper and in this paper, an understanding of underlying mechanisms coupled with characteristics of computational environments is crucial for better‐performing implementations of Hadoop‐based applications.…”

“…The different system size requires different computational workloads and affects the performance of each step along with the Hadoop‐based workflow. In the conference paper , we compared time‐to‐solutions using these two molecular systems for two different implementations for RESTMD, in order to find a right decision on the implementation. Based on results, for HaRE, we use one implementation, called I M B , with the larger system, S B , unless specifically stated otherwise.…”

“…Interestingly, the time‐to‐solution of a whole MR varies considerably depending upon the configuration of the two‐cluster system and two implementations, I M A and I M B . These two implementations were described extensively in the conference paper and, in brief, represent two possible implementation strategies that differ into what extent the local file system is utilized. Striking difference is observed between the two.…”

“…While an extension of MR has been previously applied to MD simulation trajectory analysis , we aim to utilize MR for performing simulations. In our previous works, we introduced our early implementation of MR‐based RESTMD with initial results of scalability performance, STMD performance, and the performance of STMD implemented in two different packages (CHARMM and LAMMPS) . Traditional MD implementations using MR are difficult because of the fine‐grained communication requirement imposed by MD.…”

A Novel Parallel Computation Model with Efficient Local Memory Management for Data-Intensive Applications

“…The model considers pipelining to improve the execution efficiency and to achieve the support for flexible pricing. Kondikoppa, et al [14] have presented a Replica Exchange Statistical Temperature Molecular Dynamics based on parallel tempering methodology for future scientific applications. To solve the non-optimized resource allocation problem in Hadoop, a novel Dynamic Hadoop Slot Allocation scheme is proposed in [15].…”