Direct migration of scientific computing experiments to the cloud

Srirama, Satish Narayana; Ivaništšev, Vladislav; Jakovits, Pelle; Willmore, Chris

doi:10.1109/hpcsim.2013.6641389

Cited by 3 publications

(3 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, in [14] we find interesting efforts to move desktop simulation applications to the cloud via virtualized bundled images that run in a transparent multi-tenant fashion from the end user's point of view, while minimizing costs. As previously discussed, we believe the virtualization middleware might affect performance since it does not take into account any structural characteristics of the model, which could be exploited to minimize cloudification effects or drastically affect execution times or resource consumption.…”

Section: Related Workmentioning

confidence: 99%

A Cloudification Methodology for Numerical Simulations

Caíno‐Lores

Garcı́a

Garcı́a-Carballeira

et al. 2014

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Many scientific areas make extensive use of computer simulations to study complex real-world processes. These computations are typically very resource-intensive and present scalability issues as experiments get larger, even in dedicated clusters since they are limited by their own hardware resources. Cloud computing raises as an option to move forward into the ideal unlimited scalability by providing virtually infinite resources, yet applications must be adapted to this new paradigm. We propose a generalist cloudification method based in the MapReduce paradigm to migrate numerical simulations into the cloud to provide greater scalability. We analysed its viability by applying it to a realworld simulation and running the resulting implementation on Hadoop YARN over Amazons EC2. Our tests show that the cloudified application is highly scalable and there is still a large margin to improve the theoretical model and its implementations, and also to extend it to a wider range of simulations.

show abstract

Section: Related Workmentioning

confidence: 99%

A Cloudification Methodology for Numerical Simulations

Caíno‐Lores

Garcı́a

Garcı́a-Carballeira

et al. 2014

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…In this work, a virtualization architecture is implemented by means of a web interface and a Software-as-a-Service market and development platform. Similarly, [139] proposes moving desktop simulation applications to the cloud via virtualized bundled images. These are generalist approaches that do not take into consideration the internal structure of the HPC applications, thus might not suffice for the resource-intensive computations required by HPC simulations.…”

Section: ) Infrastructure: Distributed Storage and Cloud Computingmentioning

confidence: 99%

Toward High-Performance Computing and Big Data Analytics Convergence: The Case of Spark-DIY

et al. 2019

View full text Add to dashboard Cite

Convergence between high-performance computing (HPC) and big data analytics (BDA) is currently an established research area that has spawned new opportunities for unifying the platform layer and data abstractions in these ecosystems. This work presents an architectural model that enables the interoperability of established BDA and HPC execution models, reflecting the key design features that interest both the HPC and BDA communities, and including an abstract data collection and operational model that generates a unified interface for hybrid applications. This architecture can be implemented in different ways depending on the process-and data-centric platforms of choice and the mechanisms put in place to effectively meet the requirements of the architecture. The Spark-DIY platform is introduced in the paper as a prototype implementation of the architecture proposed. It preserves the interfaces and execution environment of the popular BDA platform Apache Spark, making it compatible with any Spark-based application and tool, while providing efficient communication and kernel execution via DIY, a powerful communication pattern library built on top of MPI. Later, Spark-DIY is analyzed in terms of performance by building a representative use case from the hydrogeology domain, EnKF-HGS. This application is a clear example of how current HPC simulations are evolving toward hybrid HPC-BDA applications, integrating HPC simulations within a BDA environment. INDEX TERMS Big data analytics, high performance computing, spark, DIY, MPI.

show abstract

“…Finally, in [27] we find interesting efforts to move desktop simulation applications to the cloud via virtualized bundled images. These run in a transparent multi-tenant fashion from the end user's point of view, while minimizing costs.…”

Section: Related Workmentioning

confidence: 99%

A cloudification methodology for multidimensional analysis: Implementation and application to a railway power simulator

Caíno‐Lores

Fernández

Garcı́a-Carballeira

et al. 2015

Simulation Modelling Practice and Theory

View full text Add to dashboard Cite

Many scientific areas make extensive use of computer simulations to study complex real-world processes. These computations are typically very resource-intensive and present scalability issues as experiments get larger even in dedicated clusters, since these are limited by their own hardware resources. Cloud computing raises as an option to move forward into the ideal unlimited scalability by providing virtually infinite resources, yet applications must be adapted to this new paradigm. This process of converting and/or migrating an application and its data in order to make use of cloud computing is sometimes known as cloudifying the application. We propose a generalist cloudification method based in the MapReduce paradigm to migrate scientific simulations into the cloud to provide greater scalability. We analysed its viability by applying it to a real-world railway power consumption simulator and running the resulting implementation on Hadoop YARN over Amazon EC2. Our tests show that the cloudified application is highly scalable and there is still a large margin to improve the theoretical model and its implementations, and also to extend it to a wider range of simulations. We also propose and evaluate a multidimensional analysis tool based on the cloudified application. It generates, executes and evaluates several experiments in parallel, for the same simulation kernel. The results we obtained indicate that out methodology is suitable for resource intensive simulations and multidimensional analysis, as it improves infrastructure's utilization, efficiency and scalability when running many complex experiments.1.

show abstract

Direct migration of scientific computing experiments to the cloud

Cited by 3 publications

References 19 publications

A Cloudification Methodology for Numerical Simulations

A Cloudification Methodology for Numerical Simulations

Toward High-Performance Computing and Big Data Analytics Convergence: The Case of Spark-DIY

A cloudification methodology for multidimensional analysis: Implementation and application to a railway power simulator

Contact Info

Product

Resources

About