Adapting scientific computing problems to clouds using MapReduce

Srirama, Satish Narayana; Jakovits, Pelle; Vainikko, Eero

doi:10.1016/j.future.2011.05.025

Cited by 99 publications

(47 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Scientific experiments in fields like biology, earth sciences, or high energy physics are increasingly relying on data analysis to extract useful information from large experimental datasets, or results from large simulations [16,33]. These applications increase the importance of data-intensive computational models in HPC workloads, or the composition of different applications through workflows (e.g., simulation followed by results' analysis).…”

Section: Challenges In Hpc Schedulingmentioning

confidence: 99%

Towards understanding HPC users and systems: A NERSC case study

Rodrigo

Östberg

Elmroth

et al. 2018

Journal of Parallel and Distributed Computing

View full text Add to dashboard Cite

h i g h l i g h t s• A method to perform analysis of HPC systems' workloads is proposed including per year detailed and time evolution analyses.• A method to measure heterogeneity in job geometry is proposed.• State of workload of three reference HPC systems are presented.• Job geometry heterogeneity in queue is shown to affect wait time predictability. a r t i c l e i n f o t r a c tHigh performance computing (HPC) scheduling landscape currently faces new challenges due to the changes in the workload. Previously, HPC centers were dominated by tightly coupled MPI jobs. HPC workloads increasingly include high-throughput, data-intensive, and stream-processing applications. As a consequence, workloads are becoming more diverse at both application and job levels, posing new challenges to classical HPC schedulers. There is a need to understand the current HPC workloads and their evolution to facilitate informed future scheduling research and enable efficient scheduling in future HPC systems.In this paper, we present a methodology to characterize workloads and assess their heterogeneity, at a particular time period and its evolution over time. We apply this methodology to the workloads of three systems (Hopper, Edison, and Carver) at the National Energy Research Scientific Computing Center (NERSC). We present the resulting characterization of jobs, queues, heterogeneity, and performance that includes detailed information of a year of workload (2014) and evolution through the systems ' lifetime (2010-2014).

show abstract

Section: Challenges In Hpc Schedulingmentioning

confidence: 99%

Towards understanding HPC users and systems: A NERSC case study

Rodrigo

Östberg

Elmroth

et al. 2018

Journal of Parallel and Distributed Computing

View full text Add to dashboard Cite

show abstract

“…These applications can be legacy/complex code or very optimized solvers, which are hard to re-code or decompose. Some algorithm classes are hard to program within the MapReduce paradigm [28], thus it may be a good alternative to keep them as black boxes and optimize the whole experimental process rather than the parallel execution of the algorithm internally. In addition, the dynamic loop iterates along several chained executable codes, rather than one map reduce algorithm.…”

Section: Related Workmentioning

confidence: 99%

“…As done today, programming the dynamic iteration with big data management tools [28] may not be viable due to legacy scientific code complexity and the requirement for sophisticated provenance querying support. Dynamic loops have a data-centric iterative specification because they need to naturally respond to data-driven events.…”

Section: Introductionmentioning

confidence: 99%

Data-centric iteration in dynamic workflows

Dias

Guerra

Rochinha

et al. 2015

Future Generation Computer Systems

View full text Add to dashboard Cite

SummaryDynamic workflows are scientific workflows to support computational science simulations, typically using dynamic processes based on runtime scientific data analyses. They require the ability of adapting the workflow, at runtime, based on user input and dynamic steering. Supporting data-centric iteration is an important step towards dynamic workflows because user interaction with workflows is iterative. However, current support for iteration in scientific workflows is static and does not allow for changing data at runtime. In this paper, we propose a solution based on algebraic operators and a dynamic execution model to enable workflow adaptation based on user input and dynamic steering. We introduce the concept of iteration lineage that makes provenance data management consistent with dynamic iterative workflow changes. Lineage enables scientists to interact with workflow data and configuration at runtime through an API that triggers steering. We evaluate our approach using a novel and real large-scale workflow for uncertainty quantification on a 640-core cluster. The results show impressive execution time savings from 2.5 to 24 days, compared to noniterative workflow execution. We verify that the maximum overhead introduced by our iterative model is less than 5% of execution time. Also, our proposed steering algorithms are very efficient and run in less than 1 millisecond, in the worst-case scenario.

show abstract

“…Besides, provided that many machines may remain idle for long periods of time, the distributed computing environment can be under-used and inefficient. Previous projects [4] have already studied the scope of establishing private clouds at the universities. With these clouds, students and researches can efficiently use the already existing resources of university computer networks in solving computationally intensive scientific problems.…”

Section: Introductionmentioning

confidence: 99%

Assessment, Design and Implementation of a Private Cloud for MapReduce Applications

Salgueiro¹,

González²,

Pena³

et al. 2014

OALib

View full text Add to dashboard Cite

Scientific computation and data intensive analyses are ever more frequent. On the one hand, the MapReduce programming model has gained a lot of attention for its applicability in large parallel data analyses and Big Data applications. On the other hand, Cloud computing seems to be increasingly attractive in solving these computing problems that demand a lot of resources. This paper explores the potential symbiosis between MapReduce and Cloud Computing, in order to create a robust and scalable environment to execute MapReduce workflows regardless of the underlaying infrastructure. The main goal of this work is to provide an easy-to-install interface, so as non-expert scientists can deploy a suitable testbed for their MapReduce experiments on local resources of their institution. Testing cases were performed in order to evaluate the required time for the whole executing process on a real cluster.

show abstract

Adapting scientific computing problems to clouds using MapReduce

Cited by 99 publications

References 10 publications

Towards understanding HPC users and systems: A NERSC case study

Towards understanding HPC users and systems: A NERSC case study

Data-centric iteration in dynamic workflows

Assessment, Design and Implementation of a Private Cloud for MapReduce Applications

Contact Info

Product

Resources

About