Profile-guided I/O partitioning

Wang, Yijian; Kaeli, David

doi:10.1145/782814.782850

Cited by 40 publications

(23 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Instead of using SSD, Wang et al proposed to replicate frequently accessed data chunks at the compute nodes' local disks to reduce data access latency [28]. The frequently accessed data chunks are identified through analysis of I/O traces collected in a profiling run.…”

Section: A Approaches To Handle Unaligned Data Accessmentioning

confidence: 99%

iBridge: Improving Unaligned Parallel File Access with Solid-State Drives

Zhang

Liu

Davis³

et al. 2013

2013 IEEE 27th International Symposium on Parallel and Distributed Processing

View full text Add to dashboard Cite

Abstract-When files are striped in a parallel I/O system, requests to the files are decomposed into a number of subrequests that are distributed over multiple servers. If a request is not aligned with the striping pattern such decomposition can make the first and last sub-requests much smaller than the striping unit. Because hard-disk-based servers can be much less efficient in serving small requests than large ones, the system exhibits heterogeneity in serving sub-requests of different sizes, and the net throughput of the entire system can be severely degraded by the inefficiency of serving the smaller requests, or fragments. Because a request is not considered complete until its slowest sub-request is, the penalty is yet greater for synchronous requests. To make the situation even worse, the larger the request, or the more data servers the requested data is striped over, the larger the detrimental performance effect of serving fragments can be. This effect can become the Achilles' heel of a parallel I/O system performance seeking scalability with large sequential accesses.In this paper we propose iBridge, a scheme that uses solidstate drives to serve request fragments and thereby bridge the performance gap between serving fragments and serving large sub-requests. We have implemented iBridge in the PVFS file system. Our experimental results with representative MPI-IO benchmarks show that iBridge can significantly improve the I/O throughput of storage systems, especially for large requests with fragments.

show abstract

Section: A Approaches To Handle Unaligned Data Accessmentioning

confidence: 99%

iBridge: Improving Unaligned Parallel File Access with Solid-State Drives

Zhang

Liu

Davis³

et al. 2013

2013 IEEE 27th International Symposium on Parallel and Distributed Processing

View full text Add to dashboard Cite

show abstract

“…With data replication, some actively used data would have multiple copies on the disk and the copy that is closest to the disk head is accessed. Replication can be carried out within one disk [10], [3], [15] or across disks [33], [30]. The effectiveness of this method relies on two factors: a stable and predictable access pattern to know where to relocate or replicate data; and, a relatively small on-disk working set so that the replication overhead is not excessive.…”

Section: B Rearrangement Of On-disk Data Layout For Greater Spatial mentioning

confidence: 99%

IOrchestrator: Improving the Performance of Multi-node I/O Systems via Inter-Server Coordination

Zhang

Davis

Jiang

2010

2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

Abstract-A cluster of data servers and a parallel file system are often used to provide high-throughput I/O service to parallel programs running on a compute cluster. To exploit I/O parallelism parallel file systems stripe file data across the data servers. While this practice is effective in serving asynchronous requests, it may break individual program's spatial locality, which can seriously degrade I/O performance when the data servers concurrently serve synchronous requests from multiple I/O-intensive programs.In this paper we propose a scheme, IOrchestrator, to improve I/O performance of multi-node storage systems by orchestrating I/O services among programs when such inter-data-server coordination is dynamically determined to be cost effective. We have implemented IOrchestrator in the PVFS2 parallel file system. Our experiments with representative parallel benchmarks show that IOrchestrator can significantly improve I/O performanceby up to a factor of 2.5-delivered by a cluster of data servers servicing concurrently-running parallel programs. Notably, we have not observed any scenarios in which the use of IOrchestrator causes substantial performance degradation.

show abstract

“…These I/O characteristics prevent disk bandwidth to be fully utilized and impact I/O performance. Access patterns can be detected both statically (at compile time) [3,7,12,16] and dynamically (at runtime) [17]. In [7,16], Paek et al use Linear Memory Access Descriptors to detect memory array access patterns within loop nests.…”

Section: Introductionmentioning

confidence: 99%

“…In [7,16], Paek et al use Linear Memory Access Descriptors to detect memory array access patterns within loop nests. In our previous studies [1,17], we have used profile-directed optimization to improve both memory and disk I/O accesses. In [17], we found that disk I/O accesses exhibit very regular and highly predictable patterns.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Source level transformations to improve I/O data partitioning

Wang¹,

Kaeli²

2003

Proceedings of the International Workshop on Storage Network Architecture and Parallel I/Os - SNAPI '03

Self Cite

View full text Add to dashboard Cite

The main goal for parallel I/O is to increase I/O parallelism by providing multiple, independent data channels between processors and disks. To realize this goal, I/O streams need to be parallelized and partitioned at multiple system layers. Contention at any level can dramatically decrease performance and limit scalability. To address this disk contention bottleneck, it is important to carefully study disk access patterns.From our previous work on I/O profiling, we found that I/O access patterns of parallel scientific applications are usually very regular and highly predictable. Thus it is possible to detect I/O access patterns statically during compiler time. Large datasets are logically linearized in file space on disk, and these intensive data accesses follow a linear space traversal. In this paper, we present our recent work on compilerdirected I/O partitioning, based on Linear Disk Access Descriptors (LDAD). We use the SUIF compiler infrastructure to perform data-flow analysis and recognize LDADs. We then use these LDADs to guide our I/O data partitioning that utilizes multiple disks to significantly increase I/O throughput.

show abstract

Profile-guided I/O partitioning

Cited by 40 publications

References 23 publications

iBridge: Improving Unaligned Parallel File Access with Solid-State Drives

iBridge: Improving Unaligned Parallel File Access with Solid-State Drives

IOrchestrator: Improving the Performance of Multi-node I/O Systems via Inter-Server Coordination

Source level transformations to improve I/O data partitioning

Contact Info

Product

Resources

About