CephFS: a new generation storage platform for Australian high energy physics

borges, gabriel xavier gonçalves; Crosby, Sean C.; Boland, L.

doi:10.1088/1742-6596/898/6/062015

Cited by 7 publications

(1 citation statement)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We implement one instantiation of our design paradigm using RADOS [32] as the object-storage backend, CephFS [9] as the POSIX layer, Apache Arrow [7,29] as the data access library, and Parquet as the file format. We evaluate the performance of our implementation by varying parameters such as cluster size and parallelism and measure metrics like query latency and CPU utilization.…”

Section: Introductionmentioning

confidence: 99%

Towards an Arrow-native Storage System

Chakraborty,

Jimenez,

Rodriguez

et al. 2021

Preprint

View full text Add to dashboard Cite

With the ever-increasing dataset sizes, several file formats like Parquet, ORC, and Avro have been developed to store data efficiently and to save network and interconnect bandwidth at the price of additional CPU utilization. However, with the advent of networks supporting 25-100 Gb/s and storage devices delivering 1, 000, 000 reqs/sec the CPU has become the bottleneck, trying to keep up feeding data in and out of these fast devices. The result is that data access libraries executed on single clients are often CPU-bound and cannot utilize the scale-out benefits of distributed storage systems. One attractive solution to this problem is to offload data-reducing processing and filtering tasks to the storage layer. However, modifying legacy storage systems to support compute offloading is often tedious and requires extensive understanding of the internals. Previous approaches re-implemented functionality of data processing frameworks and access library for a particular storage system, a duplication of effort that might have to be repeated for different storage systems.In this paper, we introduce a new design paradigm that allows extending programmable object storage systems to embed existing, widely used data processing frameworks and access libraries into the storage layer with minimal modifications. In this approach data processing frameworks and

show abstract

Section: Introductionmentioning

confidence: 99%