SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence

Abstract

Cloud developers traditionally rely on purposespecific services to provide the storage model they need for an application. In contrast, HPC developers have a much more limited choice, typically restricted to a centralized parallel file system for persistent storage. Unfortunately, these systems often offer very low performance when subject to highly-concurrent, conflicting I/O patterns. This makes difficult the implementation of inherently concurrent data structures such as distributed shared logs. Yet, this data structure is key to applications such as computational steering, data collection from physical sensor grids or discrete event generators. In this paper we tackle this issue. We present SLoG, a shared log middleware providing a shared log abstraction over a parallel file system, designed to circumvent the aforementioned limitations. We evaluate SLoG design on up to 100,000 cores of the Theta supercomputer: it demonstrates high append velocity at scale while also providing substantial benefits for other persistent backend storage systems.