FreeLoader: Scavenging Desktop Storage Resources for Scientific Data

Abstract

High-end computing is suffering a data deluge from experiments, simulations, and apparatus that creates overwhelming application dataset sizes. End-user workstations-despite more processing power than ever before-are ill-equipped to cope with such data demands due to insufficient secondary storage space and I/O rates. Meanwhile, a large portion of desktop storage is unused. We present the FreeLoader framework, which aggregates unused desktop storage space and I/O bandwidth into a shared cache/scratch space, for hosting large, immutable datasets and exploiting data access locality. Our experiments show that FreeLoader is an appealing low-cost solution to storing massive datasets, by delivering higher data access rates than traditional storage facilities. In particular, we present novel data striping techniques that allow FreeLoader to efficiently aggregate a workstation's network communication bandwidth and local I/O bandwidth. In addition, the performance impact on the native workload of donor machines is small and can be effectively controlled.

Improving the availability of supercomputer job input data using temporal replication

“…27.5 Legend as in Fig. 8(a Remote reconstruction: Some of our previous studies [15,25,30] investigated approaches for reconstructing missing pieces of datasets (either due to cache miss or failures) from data sources where the job input data was originally staged from. By modifying the parallel file system to record the data source location as a file metadata item, such remote data reconstruction can be performed on demand and in a transparent fashion.…”

Improving Data Availability for Better Access Performance: A Study on Caching Scientific Data on Distributed Desktop Workstations

“…Therefore the I/O and network workload generated by recovery will not exceed those of the original data access pattern without local caching. Meanwhile, we have shown that the negative performance impact of data fetching/serving on donated storage is modest [57] and can be well controlled [52].…”

“…Our simulation-based study is motivated by, and parameterized with the FreeLoader storage aggregation prototype [57], built atop scavenged storage resources. It aggregates unused disk space in a LAN setting into a unified cache and scratch space for storing scientific datasets.…”

“…This results in better overall access throughput. These techniques have been shown to be viable in our previous work on FreeLoader [57,58], an aggregate storage infrastructure for large scientific datasets.…”

“…In this context, we are faced with two key challenges: (1) the finite amount of contributed cache space is stretched This work builds on several of our previously published studies. While we included extended introduction of the FreeLoader framework [57,58] as well as our combination of prefix caching and collective download techniques [33], this paper proposes the novel RDPR technique as well as a new striping-enabled cache management algorithm. We also discussed methods to combine RDPR with prefix caching and collective download techniques.…”