Efficient access to many small files in a filesystem for grid computing

Thain, Douglas; Moretti, Christopher

doi:10.1109/grid.2007.4354139

Cited by 18 publications

(16 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Once acknowledgement for transfer completion is received, another file transfer can be requested. This adds at least three additional round-trip-times over the control channel [8,23]. The data channel stays idle while waiting for the next transfer command to be issued.…”

Section: Climate Data Distribution Over 100gbpsmentioning

confidence: 99%

See 1 more Smart Citation

Experiences with 100Gbps network applications

Balman

Pouyoul

Yao

et al. 2012

Proceedings of the Fifth International Workshop on Data-Intensive Distributed Computing Date

View full text Add to dashboard Cite

Abstract100Gbps networking has finally arrived, and many research and educational institutions have begun to deploy 100Gbps routers and services. ESnet and Internet2 worked together to make 100Gbps networks available to researchers at the Supercomputing 2011 conference in Seattle Washington. In this paper, we describe two of the first applications to take advantage of this network. We demonstrate a visualization application that enables remotely located scientists to gain insights from large datasets. We also demonstrate climate data movement and analysis over the 100Gbps network. We describe a number of application design issues and host tuning strategies necessary for enabling applications to scale to 100Gbps rates.

show abstract

Section: Climate Data Distribution Over 100gbpsmentioning

confidence: 99%

“…This puts extra burden on filesystem access and network transfer protocols. An important challenge in dealing with climate data movement is the lots-of-small-files problem [15,23,8]. Most of the end-to-end data transfer tools are designed for moving large data files.…”

Section: Motivationmentioning

confidence: 99%

Experiences with 100Gbps network applications

Balman

Pouyoul

Yao

et al. 2012

Proceedings of the Fifth International Workshop on Data-Intensive Distributed Computing Date

View full text Add to dashboard Cite

show abstract

“…3, the client tools use iRODS API to call relevant iRODS operations at the lowest level to handle I/O requests of an application. Petashell uses an existing open-source software, Parrot [19], to catch system I/O calls of an application and to match them with the respective iRODS I/O calls. On the other hand, petafs pretends as a filesystem to handle system I/O calls and maps filesystem calls to respective the iRODS calls through a special interface called FUSE.…”

Section: Client Toolsmentioning

confidence: 99%

PetaShare: A Reliable, Efficient and Transparent Distributed Storage Management System

Kosar

Akturk

Balman

et al. 2011

Scientific Programming

View full text Add to dashboard Cite

Abstract.Modern collaborative science has placed increasing burden on data management infrastructure to handle the increasingly large data archives generated. Beside functionality, reliability and availability are also key factors in delivering a data management system that can efficiently and effectively meet the challenges posed and compounded by the unbounded increase in the size of data generated by scientific applications. We have developed a reliable and efficient distributed data storage system, PetaShare, which spans multiple institutions across the state of Louisiana. At the back-end, PetaShare provides a unified name space and efficient data movement across geographically distributed storage sites. At the front-end, it provides light-weight clients the enable easy, transparent and scalable access. In PetaShare, we have designed and implemented an asynchronously replicated multi-master metadata system for enhanced reliability and availability, and an advanced buffering system for improved data transfer performance. In this paper, we present the details of our design and implementation, show performance results, and describe our experience in developing a reliable and efficient distributed data management system for data-intensive science.

show abstract

“…In this paper, we provide a broad overview of all aspects of the Chirp distributed filesystem, building on several previous publications [44,27,43,32,47] that introduced different technical elements independently. Section 2 discusses the unique properties needed for a grid filesystem.…”

Section: Introductionmentioning

confidence: 99%

Chirp: a practical global filesystem for cluster and Grid computing

2008

Self Cite

View full text Add to dashboard Cite

Abstract. Traditional distributed filesystem technologies designed for local and campus area networks do not adapt well to wide area grid computing environments. To address this problem, we have designed the Chirp distributed filesystem, which is designed from the ground up to meet the needs of grid computing. Chirp is easily deployed without special privileges, provides strong and flexible security mechanisms, tunable consistency semantics, and clustering to increase capacity and throughput. We demonstrate that many of these features also provide order-of-magnitude performance increases over wide area networks. We describe three applications in bioinformatics, biometrics, and gamma ray physics that each employ Chirp to attack large scale data intensive problems.

show abstract

Efficient access to many small files in a filesystem for grid computing

Cited by 18 publications

References 21 publications

Experiences with 100Gbps network applications

Experiences with 100Gbps network applications

PetaShare: A Reliable, Efficient and Transparent Distributed Storage Management System

Chirp: a practical global filesystem for cluster and Grid computing

Contact Info

Product

Resources

About