A flexible I/O arbitration framework for netCDF‐based big data processing workflows on high‐end supercomputers

Abstract

Summary On the verge of the convergence between high‐performance computing and Big Data processing, it has become increasingly prevalent to deploy large‐scale data analytics workloads on high‐end supercomputers. Such applications often come in the form of complex workflows with various different components, assimilating data from scientific simulations as well as from measurements streamed from sensor networks, such as radars and satellites. For example, as part of the Flagship 2020 (post‐K) supercomputer project of Japan, RIKEN is investigating the feasibility of a highly accurate weather forecasting system that would provide a real‐time outlook for severe guerrilla rainstorms. One of the main performance bottlenecks of this application is the lack of efficient communication among workflow components, which currently takes place over the parallel file system.In this paper, we present an initial study of a direct communication framework designed for complex workflows that eliminates unnecessary file I/O among components. Specifically, we propose an I/O arbitration layer that provides direct parallel data transfer (both synchronous and asynchronous) among job components that rely on the netCDF interface for performing I/O operations. Our solution requires only minimal modifications to application code. Moreover, we propose a configuration file–based approach that allows users to specify the desired data transfer pattern among workflow components, offering a general solution for different application contexts. We present a preliminary evaluation of the proposed framework on the K Computer (running on up to 4800 compute nodes) using RIKEN's experimental weather forecasting workflow as a case study.

A global coupled ensemble data assimilation system using the Community Earth System Model and the Data Assimilation Research Testbed

“…More generally, both hardware‐ and software‐based strategies for speeding up inter‐executable communication on distributed memory computers with little or no impact on the communicating programs are emerging within the community (e.g. (Liao et al )).…”

A New Readahead Framework for SSD-based Caching Storage in IoT Systems

“…Based on this observation, sequential prefetching was proposed to optimize I/O performance, and has long been used in operating systems [15]. UNIX-based operating systems [16], database servers [17,18], distributed file systems [19], big data processing [20], cloud storage systems [21], mobile devices [22], and high-end storage controllers [23] also employ sequential prefetching. Linux has used sequential prefetching since 2002 [24][25][26].…”

A novel approach for large‐scale environmental data partitioning on cloud and on‐premises storage for compute continuum applications

“…Most of these mechanisms can also be used in processing environmental data, as both fields are expected to access the data anywhere and anytime efficiently and securely. Liao et al 25 The majority of contemporary storage formats for multidimensional arrays, such as HDF5, NetCDF, TIFF, and Zarr, provide the ability to manage the storage and compression of data; however, these options can yield notable performance consequences. The solution in Reference 34 explored different strategies for chunking the data into units of different sizes across the dataset's temporal and spatial dimensions.…”

“…Most of these mechanisms can also be used in processing environmental data, as both fields are expected to access the data anywhere and anytime efficiently and securely. Liao et al 25 presented a direct communication framework designed for complex workflows that eliminate unnecessary file I/O among components in large‐scale HPC systems. Specifically, they propose an I/O arbitration layer that provides direct parallel data transfer (both synchronous and asynchronous) among job components that rely on the netCDF interface for performing I/O operations.…”