Parallel file systems for the IBM SP computers

Corbett, Peter; Feltelson, D. G.; Prost, Jean-Pierre; Almási, George; Baylor, Sandra Johnson; Bolmarcich, Anthony; Hsu, Yarsun; Satran, J.; Snir, Marc; Colao, R.; Herr, B.; Kavaky, J.; Morgan, Tom; Ziotek, A.

doi:10.1147/sj.342.0222

Cited by 57 publications

(23 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore different accesses interleave with each other; in many cases, this is an interleav-ing of streams of strided access used to read or write multidimensional data structures [149,515,415,516,554]. This interleaving leads to a phenomenon known as interprocess locality, because accesses by one process allow the system to predict accesses by other processes [412].…”

Section: Parallel File Systemsmentioning

confidence: 99%

Workload Modeling for Computer Systems Performance Evaluation

Feitelson

2015

197

131

View full text Add to dashboard Cite

Reliable performance evaluations require the use of representative workloads. This is no easy task since modern computer systems and their workloads are complex, with many interrelated attributes and complicated structures. Experts often use sophisticated mathematics to analyze and describe workload models, making these models difficult for practitioners to grasp. This book aims to close this gap by emphasizing the intuition and the reasoning behind the definitions and derivations related to the workload models. It provides numerous examples from real production systems, with hundreds of graphs. Using this book, readers will be able to analyze collected workload data and clean it if necessary, derive statistical models that include skewed marginal distributions and correlations, and consider the need for generative models and feedback from the system. The descriptive statistics techniques covered are also useful for other domains.

show abstract

Section: Parallel File Systemsmentioning

confidence: 99%

Workload Modeling for Computer Systems Performance Evaluation

Feitelson

2015

197

131

View full text Add to dashboard Cite

show abstract

“…At a high level, collective I/O is one of the most popular techniques for improving I/O efficiency, wherein all processes, typically on separate compute nodes, cooperate to carry out large-scale I/O transactions by exploiting regular patterns in the I/O behavior of user applications. This has been widely used in parallel and distributed systems such as PIOFS [5] and GPFS [6], and run-time I/O libraries such as MPI-IO [7]. At a lower level, cooperative caching of fixed-size file blocks has been proposed to make client caches more effectively satisfy I/O requests before they reach file servers.…”

Section: Cooperative Cachingmentioning

confidence: 99%

Improving Networked File System Performance Using a Locality-Aware Cooperative Cache Protocol

Jiang

Zhang

Liang

et al. 2010

IEEE Trans. Comput.

View full text Add to dashboard Cite

Abstract-In a distributed environment the utilization of file buffer caches in different clients may greatly vary. Cooperative caching has been proposed to increase cache utilization by coordinating the shared usage of distributed caches. It allows clients that would more greatly benefit from larger caches to forward data objects to peer clients with relatively underutilized caches. To support such coordination, global cache utilization must be dynamically evaluated. This in turn requires an effective analysis of application data access patterns. Existing coordination protocols are demonstrably suboptimal in this respect, exhibiting inefficient memory utilization and undue interference among clients. We propose a locality-aware cooperative caching protocol, called LAC, that is based on analysis and manipulation of data-block reuse distance to effectively predict cache utilization and the probability of data reuse at each client. Using a dynamically adaptive synchronization technique, we keep local information up to date and consistently comparable across clients. The system is highly scalable in the sense that global coordination is achieved without centralized control. We have conducted thorough trace-driven simulation experiments to assess the performance differences between LAC and various existing protocols representative of the general class. Using a realistic and representative cost model, we show that the LAC protocol significantly and consistently outperforms existing cooperative caching protocols, demonstrating high and balanced utilization of caches across all clients. In our experiments LAC reduces block access time by up to 36%, with an average of 31%, over the system without peer cache coordination, and reduces block access time by up to 22%, with an average of 13%, over the best performer of the existing protocols.

show abstract

“…A number of groups have studied automatic detection and optimization of I/O access patterns (e.g., see [22,24,11,21] and the references therein). Others have proposed parallel file systems and I/O runtime systems that provide users/programmers with easy-to-use APIs [7,35,31,26,9]. While these systems allow users/programmers to exploit optimizations for I/O, it is still in general the user's responsibility to select which optimization to apply and determine the suitable parameters for it.…”

Section: Related Workmentioning

confidence: 99%

Discretionary caching for I/O on clusters

Vilayannur

Sivasubramaniam

Kandemir

et al. 2003

CCGrid 2003. 3rd IEEE/ACM International Symposium on Cluster Computing and the Grid, 2003. Proceedings.

View full text Add to dashboard Cite

I/O bottlenecks are already a problem in many large-scale applications that manipulate huge datasets. This problem is expected to get worse as applications get larger, and the I/O subsystem performance lags behind processor and memory speed improvements. At the same time, off-the-shelf clusters of workstations are becoming a popular platform for demanding applications due to their cost-effectiveness and widespread deployment. Caching I/O blocks is one effective way of alleviating disk latencies, and there can be multiple levels of caching on a cluster of workstations.Previous studies have shown the benefits of caching -whether it be local to a particular node, or a shared global cache across the cluster -for certain applications. However, we show that while caching is useful in some situations, it can hurt performance if we are not careful about what to cache and when to bypass the cache. This paper presents compilation techniques and runtime support to address this problem. These techniques are implemented and evaluated on an experimental Linux/Pentium cluster running a parallel file system. Our results using a diverse set of applications (scientific and commercial) demonstrate the benefits of a discretionary approach to caching for I/O subsystems on clusters, providing as much as 48% savings in overall execution time over indiscriminately caching everything in some applications.

show abstract

Parallel file systems for the IBM SP computers

Cited by 57 publications

References 26 publications

Workload Modeling for Computer Systems Performance Evaluation

Workload Modeling for Computer Systems Performance Evaluation

Improving Networked File System Performance Using a Locality-Aware Cooperative Cache Protocol

Discretionary caching for I/O on clusters

Contact Info

Product

Resources

About