CEFT: A cost-effective, fault-tolerant parallel virtual file system

Zhu, Yan; Jiang, Hong

doi:10.1016/j.jpdc.2005.10.008

Cited by 11 publications

(9 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This paper extends our previous studies presented in [13,14,16] and incorporates more experiments to evaluate our proposed approach. Based on the experimental results collected from a real cluster in production mode, this paper helps shed light on the following important design and performance issues: (1) What is the impact of resource contention on the aggregate storage throughput?…”

supporting

confidence: 55%

Exploiting redundancy to boost performance in a RAID-10 style cluster-based file system

et al. 2006

Self Cite

View full text Add to dashboard Cite

While aggregating the throughput of existing disks on cluster nodes is a cost-effective approach to alleviate the I/O bottleneck in cluster computing, this approach suffers from potential performance degradations due to contentions for shared resources on the same node between storage data processing and user task computation. This paper proposes to judiciously utilize the storage redundancy in the form of mirroring existed in a RAID-10 style file system to alleviate this performance degradation. More specifically, a heuristic scheduling algorithm is developed, motivated from the observations of a simple cluster configuration, to spatially schedule write operations on the nodes with less load among each mirroring pair. The duplication of modified data to the mirroring nodes is performed asynchronously in the background. The read performance is improved by two techniques: doubling the degree of parallelism and hot-spot skipping. A synthetic benchmark is used to evaluate these algorithms in a real cluster environment and the proposed algorithms are shown to be very effective in performance enhancement.

show abstract

supporting

confidence: 55%

Exploiting redundancy to boost performance in a RAID-10 style cluster-based file system

et al. 2006

Self Cite

View full text Add to dashboard Cite

show abstract

“…As data throughput is the most important objective of PVFS, some expensive but indispensable functions such as the concurrent control between data and metadata are not fully designed and implemented. In CEFT [6], [10], [13], [17], which is an extension of PVFS to incorporate a RAID-10-style fault tolerance and parallel I/O scheduling, the MS synchronizes concurrent updates, which can limit the overall throughput under the workload of intensive concurrent metadata updates. In Lustre [1], some low-level metadata management tasks are offloaded from the MS to object storage devices, and ongoing efforts are being made to decentralize metadata management to further improve the scalability.…”

Section: Related Work and Comparison Of Decentralization Schemesmentioning

confidence: 99%

“…R APID advances in general-purpose communication networks have motivated the deployment of inexpensive components to build competitive cluster-based storage solutions to meet the increasing demand of scalable computing [1], [2], [3], [4], [5], [6]. In the recent years, the bandwidth of these networks has been increased by two orders of magnitude [7], [8], [9], which greatly narrows the performance gap between them and the dedicated networks used in commercial storage systems.…”

Section: Introductionmentioning

confidence: 99%

HBA: Distributed Metadata Management for Large Cluster-Based Storage Systems

Zhu

Jiang

Wang

et al. 2008

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

Abstract-An efficient and distributed scheme for file mapping or file lookup is critical in decentralizing metadata management within a group of metadata servers. This paper presents a novel technique called Hierarchical Bloom Filter Arrays (HBA) to map filenames to the metadata servers holding their metadata. Two levels of probabilistic arrays, namely, the Bloom filter arrays with different levels of accuracies, are used on each metadata server. One array, with lower accuracy and representing the distribution of the entire metadata, trades accuracy for significantly reduced memory overhead, whereas the other array, with higher accuracy, caches partial distribution information and exploits the temporal locality of file access patterns. Both arrays are replicated to all metadata servers to support fast local lookups. We evaluate HBA through extensive trace-driven simulations and implementation in Linux. Simulation results show our HBA design to be highly effective and efficient in improving the performance and scalability of file systems in clusters with 1,000 to 10,000 nodes (or superclusters) and with the amount of data in the petabyte scale or higher. Our implementation indicates that HBA can reduce the metadata operation time of a single-metadata-server architecture by a factor of up to 43.9 when the system is configured with 16 metadata servers.Index Terms-Distributed file systems, file system management, metadata management, Bloom filter.

show abstract

“…Scientific applications usually need to input and output large amounts of data from secondary storage systems [5]. In order to alleviate the I/O bottleneck, cluster supercomputers usually use high-end storage servers with large capacity of main memory.…”

Section: Introductionmentioning

confidence: 99%

An Energy-Oriented Evaluation of Buffer Cache Algorithms Using Parallel I/O Workloads

Yue

Zhu

Cai

2008

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

Power consumption is an important issue for cluster supercomputers as it directly affects running cost and cooling requirements. This paper investigates the memory energy efficiency of high-end data servers used for supercomputers. Emerging memory technologies allow memory devices to dynamically adjust their power states and enable free rides by overlapping multiple DMA transfers from different I/O buses to the same memory device. To achieve maximum energy saving, the memory management on data servers needs to judiciously utilize these energy-aware devices. As we explore different management schemes under five real-world parallel I/O workloads, we find that the memory energy behavior is determined by a complex interaction among four important factors: 1) cache hit rates that may directly translate performance gain into energy saving, 2) cache populating schemes that perform buffer allocation and affect access locality at the chip level, 3) request clustering that aims to temporally align memory transfers from different buses into the same memory chips, and 4) access patterns in workloads that affect the first three factors.

show abstract

CEFT: A cost-effective, fault-tolerant parallel virtual file system

Cited by 11 publications

References 60 publications

Exploiting redundancy to boost performance in a RAID-10 style cluster-based file system

Exploiting redundancy to boost performance in a RAID-10 style cluster-based file system

HBA: Distributed Metadata Management for Large Cluster-Based Storage Systems

An Energy-Oriented Evaluation of Buffer Cache Algorithms Using Parallel I/O Workloads

Contact Info

Product

Resources

About