Data-Locality Aware Scientific Workflow Scheduling Methods in HPC Cloud Environments

Choi, Jieun; Adufu, Theodora; Kim, Chongam

doi:10.1007/s10766-016-0463-0

Cited by 17 publications

(8 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The study, however, disregards network effects, assuming that data flow between co-hosted virtual machines is equally efficient as local data access. Choi et al [39] present a mechanism for locality-aware resource management in High-Performance Computing (HPC) cloud environments, called Data-Locality Aware Workflow Scheduling (D-LAWS). Their solution condenses virtual machines and includes task parallelism through data flow into the task execution planning of a data-intensive scientific process.…”

Section: Related Workmentioning

confidence: 99%

Big Data Workflows: Locality-Aware Orchestration Using Software Containers

Corodescu

Nikolov

Khan

et al. 2021

Sensors

View full text Add to dashboard Cite

The emergence of the edge computing paradigm has shifted data processing from centralised infrastructures to heterogeneous and geographically distributed infrastructures. Therefore, data processing solutions must consider data locality to reduce the performance penalties from data transfers among remote data centres. Existing big data processing solutions provide limited support for handling data locality and are inefficient in processing small and frequent events specific to the edge environments. This article proposes a novel architecture and a proof-of-concept implementation for software container-centric big data workflow orchestration that puts data locality at the forefront. The proposed solution considers the available data locality information, leverages long-lived containers to execute workflow steps, and handles the interaction with different data sources through containers. We compare the proposed solution with Argo workflows and demonstrate a significant performance improvement in the execution speed for processing the same data units. Finally, we carry out experiments with the proposed solution under different configurations and analyze individual aspects affecting the performance of the overall solution.

show abstract

Section: Related Workmentioning

confidence: 99%

Big Data Workflows: Locality-Aware Orchestration Using Software Containers

Corodescu

Nikolov

Khan

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…In both experiments, we note patterns of unnecessary spread of tasks among nodes with both engines at times. This is something to keep in mind when working with large data batches, as it is desirable to minimize data movement between nodes 40,41 . A helpful directive to this effect in Nextflow is scratch, and in Cromwell is localization_optional.…”

Section: Scalabilitymentioning

confidence: 99%

Design considerations for workflow management systems use in production genomics research and the clinic

Ahmed

Allen

Bhat

et al. 2021

Preprint

View full text Add to dashboard Cite

Background: The changing landscape of genomics research and clinical practice has created a need for computational pipelines capable of efficiently orchestrating complex analysis stages while handling large volumes of data across heterogeneous computational environments. Workflow Management Systems (WfMSs) are the software components employed to fill this gap. Results: This work provides an approach and systematic evaluation of key features of popular bioinformatics WfMSs in use today: Nextflow, CWL, and WDL and some of their executors, along with Swift/T, a workflow manager commonly used in high-scale physics applications. We employed two use cases: a variant-calling genomic pipeline and a scalability-testing framework, where both were run locally, on an HPC cluster, and in the cloud. This allowed for evaluation of those four WfMSs in terms of language expressiveness, modularity, scalability, robustness, reproducibility, interoperability, ease of development, along with adoption and usage in research labs and healthcare settings. This article is trying to answer, "which WfMS should be chosen for a given bioinformatics application regardless of analysis type?". Conclusions: The choice of a given WfMS is a function of both its intrinsic language and engine features. Within bioinformatics, where analysts are a mix of dry and wet lab scientists, the choice is also governed by collaborations and adoption within large consortia and technical support provided by the WfMS team/community. As the community and its needs continue to evolve along with computational infrastructure, WfMSs will also evolve, especially those with permissive licenses that allow commercial use. In much the same way as the dataflow paradigm and containerization are now well understood to be very useful in bioinformatics applications, we will continue to see innovations of tools and utilities for other purposes, like big data technologies, interoperability, and provenance.

show abstract

“…() WaComM uses a hybrid approach, based on Eulerian‐Lagrangian models, implemented using a heterogeneous parallel approach . WaComM requires high performance computing capabilities to be able to provide near real‐time results,() impacting the on‐premise total cost of ownership . The operational costs derived from such expensive infrastructures could be mitigated by the GPU offloader, running the CUDA applications in dedicated remote commodity GPU accelerator servers.…”

Section: Real Use Case Scenariosmentioning

confidence: 99%

Accelerating Linux and Android applications on low‐power devices through remote GPGPU offloading

Montella

Kosta

Oro³

et al. 2017

Concurrency and Computation

View full text Add to dashboard Cite

Summary Low‐power devices are usually highly constrained in terms of CPU computing power, memory, and GPGPU resources for real‐time applications to run. In this paper, we describe RAPID, a complete framework suite for computation offloading to help low‐powered devices overcome these limitations. RAPID supports CPU and GPGPU computation offloading on Linux and Android devices. Moreover, the framework implements lightweight secure data transmission of the offloading operations. We present the architecture of the framework, showing the integration of the CPU and GPGPU offloading modules. We show by extensive experiments that the overhead introduced by the security layer is negligible. We present the first benchmark results showing that Java/Android GPGPU code offloading is possible. Finally, we show the adoption of the GPGPU offloading into BioSurveillance, a commercial real‐time face recognition application. The results show that, thanks to RAPID, BioSurveillance is being successfully adapted to run on low‐power devices. The proposed framework is highly modular and exposes a rich application programming interface to developers, making it highly versatile while hiding the complexity of the underlying networking layer.

show abstract

Data-Locality Aware Scientific Workflow Scheduling Methods in HPC Cloud Environments

Cited by 17 publications

References 10 publications

Big Data Workflows: Locality-Aware Orchestration Using Software Containers

Big Data Workflows: Locality-Aware Orchestration Using Software Containers

Design considerations for workflow management systems use in production genomics research and the clinic

Accelerating Linux and Android applications on low‐power devices through remote GPGPU offloading

Contact Info

Product

Resources

About