Storage resource manager version 2.2: design, implementation, and testing experience

Abstract

Abstract. Storage Services are crucial components of the Worldwide LHC Computing Grid Infrastructure spanning more than 200 sites and serving computing and storage resources to the High Energy Physics LHC communities. Up to tens of Petabytes of data are collected every year by the four LHC experiments at CERN. To process these large data volumes it is important to establish a protocol and a very efficient interface to the various storage solutions adopted by the WLCG sites. In this work we report on the experience acquired during the definition of the Storage Resource Manager v2.2 protocol. In particular, we focus on the study performed to enhance the interface and make it suitable for use by the WLCG communities. At the moment 5 different storage solutions implement the SRM v2.2 interface: BeStMan (LBNL), CASTOR (CERN and RAL), dCache (DESY and FNAL), DPM (CERN), and StoRM (INFN and ICTP). After a detailed inside review of the protocol, various test suites have been written identifying the most effective set of tests: the S2 test suite from CERN and the SRMTester test suite from LBNL. Such test suites have helped verifying the consistency and coherence of the proposed protocol and validating existing implementations. We conclude our work describing the results achieved. 2 IntroductionThe Worldwide LHC Computing Grid (WLCG) [1] Infrastructure is the largest Grid in the world, including about 230 sites worldwide [2]. It has been mainly established to support the 4 Large Hadron Collider (LHC) experiments at CERN. The LHC is the world's biggest machine to study the fundamental properties of sub-atomic particles and is due to start operating in 2008.The goal of the WLCG project is to establish a world-wide Grid infrastructure of computing centers to provide sufficient computational, storage and network resources to fully exploit the scientific potential of the four major experiments operating on LHC data: Alice, ATLAS, CMS and LHCb. These experiments will generate enormous amounts of data (10-15 Petabytes per year).Computing and storage services to analyze them would be implemented by a geographically distributed Data Grid.Given the variety of the storage solutions adopted by the sites collaborating in the WLCG infrastructure, it was considered important to provide an efficient and uniform Grid interface to storage and allow experiments to transparently access the data, independently of the storage implementation available at a site. This effort has given rise to the Grid Storage Management Working Group (GSM-WG) at the Open Grid Forum (OGF) [3].In what follows, we report on the experience acquired during the definition of the Storage Resource Manager (SRM) v2.2 protocol. In particular, we focus on the study performed to enhance the interface and make it suitable for use by the WLCG communities.In Section 2, we elaborate on the protocol definition process and on the collection of the requirements as described by the LHC experiments. In Section 3, we talk about version 2.2 of the SRM protocol as it is defined today a...

What Goes Up, Must Go Down: A Case Study From RAL on Shrinking an Existing Storage Service

“…Storage nodes are owned by a given stager and cannot be shared. Also associated with the stager services are the processes that provide SRM [15] and xrootd [16] interfaces for external users. The CASTOR SRM also has an associated databasethis is co-located with the stager.…”

Design and Execution of make-like, distributed Analyses based on Spotify’s Pipelining Package Luigi

“…We created a Luigi file system implementation and corresponding target classes that cope with most of the storage back-ends that are deployed in the current high-energy physics landscape. It is based on Python bindings of the Grid File Access Library (GFAL) [4], which uses plugins to provide support for various back-ends, such as dCache, SRM, GridFTP, XRootD, Amazon S3, WebDAV, and Dropbox [5,6,7,8]. The implemented interface resembles the local file target implementation, extended by convenience methods for handling file transfers.…”

File Access Optimization with the Lustre Filesystem at Florida CMS T2

“…The Florida CMS Tier2 storage consists of the SRM [5] (storage resource manager) server, the GridFtp [6] servers, the Xrootd [7] servers, and the backend Lustre filesystem. The BestMan implementation is used for the SRM with a single server.…”