A Fully Decentralized File System Cache for the CernVM-FS

Blomer, Jakob; Fuhrmann, Thomas

doi:10.1109/icccn.2010.5560054

Cited by 20 publications

(18 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Deploying the "general purpose" stack was not trivial, as it is distributed using CVMFS [3,5], whose standard mode of operation requires node-local storage, which Niagara does not have. The stack was deployed using CVMFS's 'alien cache' technology [4] that makes it possible to run CVMFS on systems without local disks on each node and also leverages the performance available from the large GPFS file system. This setup means that users can experience slight lags in updates to the stack, depending on the frequency of pulls from the main repository, but in practice this only has a small impact.…”

Section: Software Stackmentioning

confidence: 99%

Deploying a Top-100 Supercomputer for Large Parallel Workloads

Ponce

Zon

Northrup

et al. 2019

Proceedings of the Practice and Experience in Advanced Research Computing on Rise of the Machines (Learning)

150

View full text Add to dashboard Cite

Niagara is currently the fastest supercomputer accessible to academics in Canada. It was deployed at the beginning of 2018 and has been serving the research community ever since. This homogeneous 60,000-core cluster, owned by the University of Toronto and operated by SciNet, was intended to enable large parallel jobs and has a measured performance of 3.02 petaflops, debuting at #53 in the June 2018 TOP500 list. It was designed to optimize throughput of a range of scientific codes running at scale, energy efficiency, and network and storage performance and capacity. It replaced two systems that SciNet operated for over 8 years, the Tightly Coupled System (TCS) and the General Purpose Cluster (GPC) [13]. In this paper we describe the transition process from these two systems, the procurement and deployment processes, as well as the unique features that make Niagara a one-of-a-kind machine in Canada.

show abstract

Section: Software Stackmentioning

confidence: 99%

Deploying a Top-100 Supercomputer for Large Parallel Workloads

Ponce

Zon

Northrup

et al. 2019

Proceedings of the Practice and Experience in Advanced Research Computing on Rise of the Machines (Learning)

150

View full text Add to dashboard Cite

show abstract

“…This allows to take advantage of ROOT just-in-time C++ compilation capabilities and direct usage of all of the existing C++ libraries developed by analyzers and experiments. The software in the Spark driver and Spark workers is distributed with CVMFS [8] in order to achieve an identical environment. The user and experiment data is located on the EOS cloud storage.…”

Section: Sparkrootmentioning

confidence: 99%

Expressing Parallelism with ROOT

Piparo

Tejedor

Guiraud

et al. 2017

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

“…To that end, we proposed in earlier work a distributed hash table (DHT) where peers can decide based on their load if they keep a file [6]. Here, in this paper, we detail that idea and present performance results.…”

Section: Distributed Memory Cachementioning

confidence: 99%

CernVM-FS

Blomer

Bunčić²,

Fuhrmann

2011

Proceedings of the First International Workshop on Network-Aware Data Management

Self Cite

View full text Add to dashboard Cite

The computing facilities used to process data for the experiments at the Large Hadron Collider at CERN are scattered around the world. The embarrassingly parallel workload allows for use of various computing resources, such as Grid sites of the Worldwide LHC Computing Grid, commercial and institutional cloud resources, as well as individual home PCs in "volunteer clouds". Unlike data, the experiment software cannot be easily split into small work units. Efficient delivery of the complex and frequently changing experiment software is a crucial step to harness heterogeneous resources.Here we present an approach to deliver software on demand using a scalable hierarchy of standard HTTP caches. We show how to tackle this problem by pre-processing software into content-addressable storage. On the worker nodes, we use a specially crafted file system that ensures data integrity and provides fault-tolerance. We show performance figures from large-scale deployment. For the most common case of computing clusters with 10 to 1000 worker nodes, we present a novel state dissemination protocol to support a fully decentralized and distributed memory cache.

show abstract

A Fully Decentralized File System Cache for the CernVM-FS

Cited by 20 publications

References 9 publications

Deploying a Top-100 Supercomputer for Large Parallel Workloads

Deploying a Top-100 Supercomputer for Large Parallel Workloads

Expressing Parallelism with ROOT

CernVM-FS

Contact Info

Product

Resources

About