Workload characterization of a leadership class storage cluster

Kim, Young-Jae; Gunasekaran, Raghul; Shipman, Galen; Dillow, David; Zhang, Zhe; Settlemyer, Bradley W.

doi:10.1109/pdsw.2010.5668066

Cited by 66 publications

(33 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, it has been found that HPC file systems are stressed with write requests of frequent and periodic checkpointing and journaling operations [29]. In our study of HPC I/O workload characterization of the Spider storage system at Oak Ridge National Laboratory, we observed that the bandwidth distributions are heavily long-tailed [22].…”

Section: Introductionmentioning

confidence: 76%

“…Under this scenario the queue-waiting time of incoming requests will increase. Server-centric enterprise data center and high-performance computing (HPC) environment workloads often have bursts of requests with low interarrival time [22], [11]. Examples of enterprise workloads that exhibit this behavior include online-transaction processing applications, such as OLTP and OLAP [3], [23].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A semi-preemptive garbage collector for solid state drives

Lee

Kim

Shipman

et al. 2011

(Ieee Ispass) Ieee International Symposium on Performance Analysis of Systems and Software

Self Cite

View full text Add to dashboard Cite

Abstract-NAND flash memory is a preferred storage media for various platforms ranging from embedded systems to enterprise-scale systems. Flash devices do not have any mechanical moving parts and provide low-latency access. They also require less power compared to rotating media. Unlike hard disks, flash devices use out-of-update operations and they require a garbage collection (GC) process to reclaim invalid pages to create free blocks. This GC process is a major cause of performance degradation when running concurrently with other I/O operations as internal bandwidth is consumed to reclaim these invalid pages. The invocation of the GC process is generally governed by a low watermark on free blocks and other internal device metrics that different workloads meet at different intervals. This results in I/O performance that is highly dependent on workload characteristics. In this paper, we examine the GC process and propose a semi-preemptive GC scheme that can preempt on-going GC processing and service pending I/O requests in the queue. Moreover, we further enhance flash performance by pipelining internal GC operations and merge them with pending I/O requests whenever possible. Our experimental evaluation of this semi-preemptive GC sheme with realistic workloads demonstrate both improved performance and reduced performance variability. Write-dominant workloads show up to a 66.56% improvement in average response time with a 83.30% reduced variance in response time compared to the non-preemptive GC scheme.

show abstract

Section: Introductionmentioning

confidence: 76%

Section: Introductionmentioning

confidence: 99%

A semi-preemptive garbage collector for solid state drives

Lee

Kim

Shipman

et al. 2011

(Ieee Ispass) Ieee International Symposium on Performance Analysis of Systems and Software

Self Cite

View full text Add to dashboard Cite

show abstract

“…Accordingly, there is a large body of work on the collection, analysis, and characterization of the workloads on storage systems, including enterprise computing environments [2,20,21] and high-performance computing environments [11,22,30]. The observations can be of great importance to system design, engineering, and tuning.…”

Section: Related Workmentioning

confidence: 99%

Workload analysis of a large-scale key-value store

AtikogluBerk¹,

XuYuehai

Frachtenberg

et al. 2012

SIGMETRICS Perform. Eval. Rev.

262

View full text Add to dashboard Cite

Key-value stores are a vital component in many scale-out enterprises, including social networks, online retail, and risk analysis. Accordingly, they are receiving increased attention from the research community in an effort to improve their performance, scalability, reliability, cost, and power consumption. To be effective, such efforts require a detailed understanding of realistic key-value workloads. And yet little is known about these workloads outside of the companies that operate them. This paper aims to address this gap.To this end, we have collected detailed traces from Facebook's Memcached deployment, arguably the world's largest. The traces capture over 284 billion requests from five different Memcached use cases over several days. We analyze the workloads from multiple angles, including: request composition, size, and rate; cache efficacy; temporal patterns; and application use cases. We also propose a simple model of the most representative trace to enable the generation of more realistic synthetic workloads by the community.Our analysis details many characteristics of the caching workload. It also reveals a number of surprises: a GET/SET ratio of 30:1 that is higher than assumed in the literature; some applications of Memcached behave more like persistent storage than a cache; strong locality metrics, such as keys accessed many millions of times a day, do not always suffice for a high hit rate; and there is still room for efficiency and hit rate improvements in Memcached's implementation. Toward the last point, we make several suggestions that address the exposed deficiencies.

show abstract

“…The I/O size of IOR is set to 1 MB, which is common among HPC systems [19]. It also matches Lustre's default I/O size.…”

Section: B Hardware and Workloads Used For Evaluationmentioning

confidence: 99%

ASCAR: Automating contention management for high-performance storage systems

Miller

et al. 2015

2015 31st Symposium on Mass Storage Systems and Technologies (MSST)

View full text Add to dashboard Cite

Abstract-High-performance parallel storage systems, such as those used by supercomputers and data centers, can suffer from performance degradation when a large number of clients are contending for limited resources, like bandwidth. These contentions lower the efficiency of the system and cause unwanted speed variances. We present the Automatic Storage Contention Alleviation and Reduction system (ASCAR), a storage traffic management system for improving the bandwidth utilization and fairness of resource allocation. ASCAR regulates I/O traffic from the clients using a rule based algorithm that controls the congestion window and rate limit. The rule-based client controllers are fast responding to burst I/O because no runtime coordination between clients or with a central coordinator is needed; they are also autonomous so the system has no scale-out bottleneck. Finding optimal rules can be a challenging task that requires expertise and numerous experiments. ASCAR includes a SHAred-nothing Rule Producer (SHARP) that produces rules in an unsupervised manner by systematically exploring the solution space of possible rule designs and evaluating the target workload under the candidate rule sets. Evaluation shows that our ASCAR prototype can improve the throughput of all tested workloadssome by as much as 35%. ASCAR improves the throughput of a NASA NPB BTIO checkpoint workload by 33.5% and reduces its speed variance by 55.4% at the same time. The optimization time and controller overhead are unrelated to the scale of the system; thus, it has the potential to support future large-scale systems that can have millions of clients and thousands of servers. As a pure client-side solution, ASCAR needs no change to either the hardware or server software.

show abstract

Workload characterization of a leadership class storage cluster

Cited by 66 publications

References 6 publications

A semi-preemptive garbage collector for solid state drives

A semi-preemptive garbage collector for solid state drives

Workload analysis of a large-scale key-value store

ASCAR: Automating contention management for high-performance storage systems

Contact Info

Product

Resources

About