Could Blobs Fuel Storage-Based Convergence Between HPC and Big Data?

Matri, Pierre; Alforov, Yevhen; Brandon, Alvaro; Kuhn, Michael; Carns, Philip; Ludwig, Thomas

doi:10.1109/cluster.2017.63

Cited by 5 publications

(4 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Given the workflow nature of many state-of-the-art simulators for scientific computing, Srirama et al [123] proposed a workflow-partitioning strategy to reduce the data communication in the resulting deployment. Matri et al [124], [125] analyzed the applicability of binary large objects (known as blobs) and object storage systems to solve the problems with POSIX-IO-compliant file systems and as a mechanism to replace distributed file systems for BDA analytics.…”

Section: ) Infrastructure: Distributed Storage and Cloud Computingmentioning

confidence: 99%

Toward High-Performance Computing and Big Data Analytics Convergence: The Case of Spark-DIY

et al. 2019

View full text Add to dashboard Cite

Convergence between high-performance computing (HPC) and big data analytics (BDA) is currently an established research area that has spawned new opportunities for unifying the platform layer and data abstractions in these ecosystems. This work presents an architectural model that enables the interoperability of established BDA and HPC execution models, reflecting the key design features that interest both the HPC and BDA communities, and including an abstract data collection and operational model that generates a unified interface for hybrid applications. This architecture can be implemented in different ways depending on the process-and data-centric platforms of choice and the mechanisms put in place to effectively meet the requirements of the architecture. The Spark-DIY platform is introduced in the paper as a prototype implementation of the architecture proposed. It preserves the interfaces and execution environment of the popular BDA platform Apache Spark, making it compatible with any Spark-based application and tool, while providing efficient communication and kernel execution via DIY, a powerful communication pattern library built on top of MPI. Later, Spark-DIY is analyzed in terms of performance by building a representative use case from the hydrogeology domain, EnKF-HGS. This application is a clear example of how current HPC simulations are evolving toward hybrid HPC-BDA applications, integrating HPC simulations within a BDA environment. INDEX TERMS Big data analytics, high performance computing, spark, DIY, MPI.

show abstract

Section: ) Infrastructure: Distributed Storage and Cloud Computingmentioning

confidence: 99%

Toward High-Performance Computing and Big Data Analytics Convergence: The Case of Spark-DIY

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Týr is a blob storage system with support for transactions; it provides blob storage functionality and high access parallelism [64]. Measurements show that many applications do not require most of the functionality provided by full-fledged file systems but instead only use a subset that can be provided by blob or object storage systems [63].…”

Section: Storage and File Systemsmentioning

confidence: 99%

Survey of Storage Systems for High-Performance Computing

Lüttgau

Kuhn

Duwe

et al. 2018

JSFI

View full text Add to dashboard Cite

In current supercomputers, storage is typically provided by parallel distributed file systems for hot data and tape archives for cold data. These file systems are often compatible with local file systems due to their use of the POSIX interface and semantics, which eases development and debugging because applications can easily run both on workstations and supercomputers. There is a wide variety of file systems to choose from, each tuned for different use cases and implementing different optimizations. However, the overall application performance is often held back by I/O bottlenecks due to insufficient performance of file systems or I/O libraries for highly parallel workloads. Performance problems are dealt with using novel storage hardware technologies as well as alternative I/O semantics and interfaces. These approaches have to be integrated into the storage stack seamlessly to make them convenient to use. Upcoming storage systems abandon the traditional POSIX interface and semantics in favor of alternative concepts such as object and key-value storage; moreover, they heavily rely on technologies such as NVM and burst buffers to improve performance. Additional tiers of storage hardware will increase the importance of hierarchical storage management. Many of these changes will be disruptive and require application developers to rethink their approaches to data management and I/O. A thorough understanding of today's storage infrastructures, including their strengths and weaknesses, is crucially important for designing and implementing scalable storage systems suitable for demands of exascale computing.

show abstract

“…It needs at the same time to evade the very limited write concurrency permitted by these storage systems, while at the same time abstracting this complexity away from the developer. Finally, when other storage models are available (e.g., object storage [29]), it should take advantage of their different capabilities to further enable performance without requiring any application modification.…”

Section: The Case For a Shared Log Middlewarementioning

confidence: 99%

SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence

Matri

Carns

Ross

et al. 2018

2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS)

Self Cite

View full text Add to dashboard Cite

Cloud developers traditionally rely on purposespecific services to provide the storage model they need for an application. In contrast, HPC developers have a much more limited choice, typically restricted to a centralized parallel file system for persistent storage. Unfortunately, these systems often offer very low performance when subject to highly-concurrent, conflicting I/O patterns. This makes difficult the implementation of inherently concurrent data structures such as distributed shared logs. Yet, this data structure is key to applications such as computational steering, data collection from physical sensor grids or discrete event generators. In this paper we tackle this issue. We present SLoG, a shared log middleware providing a shared log abstraction over a parallel file system, designed to circumvent the aforementioned limitations. We evaluate SLoG design on up to 100,000 cores of the Theta supercomputer: it demonstrates high append velocity at scale while also providing substantial benefits for other persistent backend storage systems.

show abstract

Could Blobs Fuel Storage-Based Convergence Between HPC and Big Data?

Cited by 5 publications

References 30 publications

Toward High-Performance Computing and Big Data Analytics Convergence: The Case of Spark-DIY

Toward High-Performance Computing and Big Data Analytics Convergence: The Case of Spark-DIY

Survey of Storage Systems for High-Performance Computing

SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence

Contact Info

Product

Resources

About