Developing eThread Pipeline Using SAGA-Pilot Abstraction for Large-Scale Structural Bioinformatics

Ragothaman, Anjani; Boddu, Sairam Chowdary; Kim, Nayong; Feinstein, Wei P.; Bryliński, Michał; Jha, Shantenu; Kim, Joohyun

doi:10.1155/2014/348725

Cited by 7 publications

(4 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Because of this, processing is increasingly moving to grid-and cloud-based distributed computing facilities. From genetic sequencing [11] and bioinformatics [12] to neuroscience [13] and ecology [14], ever growing datasets have driven the development of distributed workflow systems in science [15] [16].…”

Section: Related Workmentioning

confidence: 99%

An Automated Scalable Framework for Distributing Radio Astronomy Processing Across Clusters and Clouds

Mechev¹,

Oonk²,

Danezi³

et al. 2017

Proceedings of International Symposium on Grids and Clouds (ISGC) 2017 — PoS(ISGC2017)

View full text Add to dashboard Cite

The Low Frequency Array (LOFAR) radio telescope is an international aperture synthesis radio telescope used to study the Universe at low frequencies. One of the goals of the LOFAR telescope is to conduct deep wide-field surveys. Here we will discuss a framework for the processing of the LOFAR Two Meter Sky Survey (LoTSS). This survey will produce close to 50 PB of data within five years. These data rates require processing at locations with high-speed access to the archived data.To complete the LoTSS project, the processing software needs to be made portable and moved to clusters with a high bandwidth connection to the data archive. This work presents a framework that makes the LOFAR software portable, and is used to scale out LOFAR data reduction. Previous work was successful in pre-processing LOFAR data on a cluster of isolated nodes. This framework builds upon it and and is currently operational. It is designed to be portable, scalable, automated and general. This paper describes its design and high level operation and the initial results processing LoTSS data.

show abstract

Section: Related Workmentioning

confidence: 99%

An Automated Scalable Framework for Distributing Radio Astronomy Processing Across Clusters and Clouds

Mechev¹,

Oonk²,

Danezi³

et al. 2017

Proceedings of International Symposium on Grids and Clouds (ISGC) 2017 — PoS(ISGC2017)

View full text Add to dashboard Cite

show abstract

“…This requires adding data-intensive analysis while preserving high-performance computing capabilities. Having established the potential of the Pilot-Abstraction for a range of high-performance applications [36], [37], [38], we use it as the starting point for integrated high-performance compute and data-intensive analysis. We propose several extensions to RADICAL-Pilot to facilitate the integrated use of HPC and Hadoop frameworks using the Pilot-Abstraction.…”

Section: Integrating Hadoop and Spark With Radical-pilotmentioning

confidence: 99%

Hadoop on HPC: Integrating Hadoop and Pilot-Based Dynamic Resource Management

Luckow

Paraskevakos

Chantzialexiou

et al. 2016

2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)

Self Cite

View full text Add to dashboard Cite

Abstract-High-performance computing platforms such as "supercomputers" have traditionally been designed to meet the compute demands of scientific applications. Consequently, they have been architected as net producers and not consumers of data. The Apache Hadoop ecosystem has evolved to meet the requirements of data processing applications and has addressed many of the traditional limitations of HPC platforms. There exist a class of scientific applications however, that need the collective capabilities of traditional high-performance computing environments and the Apache Hadoop ecosystem. For example, the scientific domains of bio-molecular dynamics, genomics and network science need to couple traditional computing with Hadoop/Spark based analysis. We investigate the critical question of how to present the capabilities of both computing environments to such scientific applications. Whereas this questions needs answers at multiple levels, we focus on the design of resource management middleware that might support the needs of both. We propose extensions to the Pilot-Abstraction so as to provide a unifying resource management layer. This is an important step towards interoperable use of HPC and Hadoop/Spark. It also allows applications to integrate HPC stages (e. g. simulations) to data analytics. Many supercomputing centers have started to officially support Hadoop environments, either in a dedicated environment or in hybrid deployments using tools such as myHadoop. This typically involves many intrinsic, environment-specific details that need to be mastered, and often swamp conceptual issues like: How best to couple HPC and Hadoop application stages? How to explore runtime trade-offs (data localities vs. data movement)? This paper provides both conceptual understanding and practical solutions to the integrated use of HPC and Hadoop environments. Our experiments are performed on state-of-the-art production HPC environments and provide middleware for multiple domain sciences.

show abstract

“…On the other hand, the four distributed infrastructures, Amazon EC2, cyberinfrastructure for reconfigurable optical networks (CRON), global environment for network innovations (GENI), and CloudLab, are distinctly different. Amazon EC2 is a compute‐on‐demand infrastructure as a service cloud environment that is recently widely utilized for research and science gateways . CRON is our testbed system that can be configured to emulate a distributed computing infrastructure connected by configurable networking conditions.…”

Section: Hadoop‐based Replica Exchangementioning

confidence: 99%

Hadoop‐based replica exchange over heterogeneous distributed cyberinfrastructures

Platania

Shams

Chiu

et al. 2016

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

Summary We present Hadoop‐based replica exchange (HaRE), a Hadoop‐based implementation of the replica exchange scheme developed primarily for replica exchange statistical temperature molecular dynamics, an example of a large‐scale, advanced sampling molecular dynamics simulation. By using Hadoop as a framework and the MapReduce model for driving replica exchange, an efficient task‐level parallelism is introduced to replica exchange statistical temperature molecular dynamics simulations. In order to demonstrate this, we investigate the performance of our application over various distributed cyberinfrastructures (DCI), including several high‐performance computing systems, our cyberinfrastructure for reconfigurable optical networks testbed, the global environment for network innovations testbed, and the CloudLab testbed. Scalability performance analysis is shown in terms of scale‐out and scale‐up over a single high‐performance computing cluster, EC2, and CloudLab and scale‐across with cyberinfrastructure for reconfigurable optical networks and global environment for network innovations. As a result, we demonstrate that HaRE is capable of efficient execution over both homogeneous and heterogeneous DCI of varying size and configuration. Contributing factors to performance are discussed in order to provide insight towards the effects of computing environment on the execution of HaRE. With these contributions, we propose that similar loosely coupled scientific applications can also take advantage of the scalable, task‐level parallelism Hadoop MapReduce provides over various DCI. Copyright © 2016 John Wiley & Sons, Ltd.

show abstract

Developing eThread Pipeline Using SAGA-Pilot Abstraction for Large-Scale Structural Bioinformatics

Cited by 7 publications

References 34 publications

An Automated Scalable Framework for Distributing Radio Astronomy Processing Across Clusters and Clouds

An Automated Scalable Framework for Distributing Radio Astronomy Processing Across Clusters and Clouds

Hadoop on HPC: Integrating Hadoop and Pilot-Based Dynamic Resource Management

Hadoop‐based replica exchange over heterogeneous distributed cyberinfrastructures

Contact Info

Product

Resources

About