Big data analytics on traditional HPC infrastructure using two-level storage

Xuan, Pengfei; Denton, Jeffrey; Srimani, Pradip K.; Ge, Rong; Luo, Feng

doi:10.1145/2831244.2831253

Cited by 13 publications

(5 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“… 38 Traditionally, high-performance computing required specialized on-premises infrastructure with powerful hardware and dedicated resources. 39 Now, cloud computing can provide the high-performance computing and storage capabilities needed for processing such big data, 40 benefiting DEWS 4.0 by enabling efficient and timely data analysis.…”

Section: Interconnected Framework Modulesmentioning

confidence: 99%

A transformative framework reshaping sustainable drought risk management through advanced early warning systems

Masupha,

Moeletsi,

Tsubo

2024

iScience

View full text Add to dashboard Cite

Section: Interconnected Framework Modulesmentioning

confidence: 99%

A transformative framework reshaping sustainable drought risk management through advanced early warning systems

Masupha,

Moeletsi,

Tsubo

2024

iScience

View full text Add to dashboard Cite

“…Results indicated that BDA platforms still suffered from the reduced locality offered by such a setting. Consequently, the authors in [105] proposed a two-layer storage system that exploits PFS performance but incorporates an intermediate in-memory storage system, with good results.…”

Section: ) Infrastructure: Networking and Acceleratorsmentioning

confidence: 99%

Toward High-Performance Computing and Big Data Analytics Convergence: The Case of Spark-DIY

et al. 2019

View full text Add to dashboard Cite

Convergence between high-performance computing (HPC) and big data analytics (BDA) is currently an established research area that has spawned new opportunities for unifying the platform layer and data abstractions in these ecosystems. This work presents an architectural model that enables the interoperability of established BDA and HPC execution models, reflecting the key design features that interest both the HPC and BDA communities, and including an abstract data collection and operational model that generates a unified interface for hybrid applications. This architecture can be implemented in different ways depending on the process-and data-centric platforms of choice and the mechanisms put in place to effectively meet the requirements of the architecture. The Spark-DIY platform is introduced in the paper as a prototype implementation of the architecture proposed. It preserves the interfaces and execution environment of the popular BDA platform Apache Spark, making it compatible with any Spark-based application and tool, while providing efficient communication and kernel execution via DIY, a powerful communication pattern library built on top of MPI. Later, Spark-DIY is analyzed in terms of performance by building a representative use case from the hydrogeology domain, EnKF-HGS. This application is a clear example of how current HPC simulations are evolving toward hybrid HPC-BDA applications, integrating HPC simulations within a BDA environment. INDEX TERMS Big data analytics, high performance computing, spark, DIY, MPI.

show abstract

“…We made the choice to use a DFS to avoid the traditional HPC IO bottleneck and scale linearly in terms of bandwidth [20], without adding expensive hardware to scale up the PFS [8].…”

Section: Related Workmentioning

confidence: 99%

Big data and HPC collocation: Using HPC idle resources for Big Data analytics

Mercier

Glesser

Georgiou

et al. 2017

2017 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Abstract-Executing Big Data workloads upon High Performance Computing (HPC) infrastractures has become an attractive way to improve their performances. However, the collocation of HPC and Big Data workloads is not an easy task, mainly because of their core concepts' differences. This paper focuses on the challenges related to the scheduling of both Big Data and HPC workloads on the same computing platform.In classic HPC workloads, the rigidity of jobs tends to create holes in the schedule: we can use those idle resources as a dynamic pool for Big Data workloads. We propose a new idea based on Resource and Job Management System's (RJMS) configuration, that makes HPC and Big Data systems to communicate through a simple prolog/epilog mechanism. It leverages the built-in resilience of Big Data frameworks, while minimizing the disturbance on HPC workloads.We present the first study of this approach, using the production RJMS middleware OAR and Hadoop YARN from the HPC and Big Data ecosystems respectively. Our new technique is evaluated with real experiments upon the Grid5000 platform. Our experiments validate our assumptions and show promising results. The system is capable of running an HPC workload with 70% cluster utilization, with a Big Data workload that fills the schedule holes to reach a full 100% utilization. We observe a penalty on the mean waiting time for HPC jobs of less than 17% and a Big Data effectiveness of more than 68% in average.

show abstract

Big data analytics on traditional HPC infrastructure using two-level storage

Cited by 13 publications

References 18 publications

A transformative framework reshaping sustainable drought risk management through advanced early warning systems

A transformative framework reshaping sustainable drought risk management through advanced early warning systems

Toward High-Performance Computing and Big Data Analytics Convergence: The Case of Spark-DIY

Big data and HPC collocation: Using HPC idle resources for Big Data analytics

Contact Info

Product

Resources

About