CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data

Abstract

Emerging large-scale distributed storage systems are faced with the task of distributing petabytes of data among tens or hundreds of thousands of storage devices. Such systems must evenly distribute data and workload to efficiently utilize available resources and maximize system performance, while facilitating system growth and managing hardware failures. We have developed CRUSH, a scalable pseudorandom data distribution function designed for distributed object-based storage systems that efficiently maps data objects to storage devices without relying on a central directory. Because large systems are inherently dynamic, CRUSH is designed to facilitate the addition and removal of storage while minimizing unnecessary data movement. The algorithm accommodates a wide variety of data replication and reliability mechanisms and distributes data in terms of userdefined policies that enforce separation of replicas across failure domains.

Content Espresso: A Distributed Large File Sharing System for Digital Content Productions

“…In order to avoid it, Ceph has a high reliable centralized metadata server cluster and adopts Dynamic Subtree Partitioning [36] but preparing a lot of high reliable servers causes high cost. While Content Espresso stores chunks by striping to Chunk Servers, Ceph utilizes CRUSH [37], which reduces object data movement and achieves scalability by appending new OSDs. When new Chunk Servers are appended in Content Espresso, it is not necessary to move stored chunks to the new Chunk Servers because Content Espresso stores chunks evenly to all of the Chunk Servers.…”

The deployment of a large scale object store at the RAL Tier-1

“…In Ceph, the CRUSH Map [4] describes the layout of the storage cluster. The CRUSH ruleset is a method to select which disks make up a placement group.…”

Impact of Single Parameter Changes on Ceph Cloud Storage Performance

“…Clients interact with the metadata server to perform operations, such as open and rename, while communicating directly with the OSDs for I/O operations. The algorithm that is used to spread the data over the available OSDs is called CRUSH [18]. From a high level, Ceph clients and metadata servers view the object storage cluster, that consists of possibly tens or hundreds of thousands of OSDs, as a single logical object store and namespace.…”