Leveraging Naturally Distributed Data Redundancy to Reduce Collective I/O Replication Overhead

Abstract

Dumping large amounts of related data simultaneously to local storage devices instead of a parallel file system is a frequent I/O pattern of HPC applications running at large scale. Since local storage resources are prone to failures and have limited potential to serve multiple requests in parallel, techniques such as replication are often used to enable resilience and high availability. However, replication introduces overhead, both in terms of network traffic necessary to distribute replicas, as well as extra storage space requirements. To reduce this overhead, state-of-art techniques often apply redundancy elimination (e.g. compression or deduplication) before replication, ignoring the natural redundancy that is already present. By contrast, this paper proposes a novel scheme that treats redundancy elimination and replication as a single co-optimized phase: remotely duplicated data is detected and directly leveraged to maintain a desired replication factor by keeping only as many replicas as needed and adding more if necessary. In this context, we introduce a series of high performance algorithms specifically designed to operate under tight and controllable constrains at large scale. We present how this idea can be leveraged in practice and demonstrate its viability for two real-life HPC applications.

A data replication strategy with tenant performance and provider economic profit guarantees in Cloud data centers

“…(ii) Some data replication strategies are based on techniques that achieve specific tenant SLOs. Examples of these techniques are: compression (Liu and Shen, 2016a) for data durability, or for reducing replication bandwidth (Xu et al, 2015), multi-failure resilient scheme (Liu and Shen, 2016b) for enhancing availability, de-duplication (Nicolae, 2015) for reducing data transfer, prefetching (Mansouri and Javidi, 2018), data migration (Mansouri and Buyya, 2019), parallel downloading (Mansouri et al, 2017), data mining (Hamrouni and Charrada, 2015), supervised learning (Bui et al, 2016), overheating similarity of nodes (Sun et al, 2018), partitioning (Zhou and Fan, 2017) for ensuring performance and fragmentation for optimal security (Ali et al, 2018). On the other hand, many corporations, e.g., Facebook, as well as many replication strategies (Bui et al, 2016) are based on the erasure coding technique rather than/in addition to data replication.…”

DataStates: Towards Lightweight Data Models for Deep Learning

“…VELOC [24,31] takes this approach further by introducing asynchronous techniques to apply such complementary strategies in the background. When the checkpoints of different processes have similar content, techniques such as [20,21] can be applied to complement multi-level checkpointing. However, redundancy is detected onthe-fly, which can be an unnecessary overhead for clone and revisit (e.g., model replicas are known to be identical for data-parallel training).…”

Exploring Shared State in Key-Value Store for Window-Based Multi-pattern Streaming Analytics

“…Deduplication is a common technique used in a variety of scenarios, both obvious (e.g., saving space in file systems [22], [11] or reducing the size of large scale memory dumps [18]) and less obvious (e.g. detection of natural replicas to reduce the cost of replication-based resilience [19]).…”