dCache: Big Data storage for HEP communities and beyond

Abstract

Abstract. With over ten years in production use dCache data storage system has evolved to match ever changing lansdcape of continually evolving storage technologies with new solutions to both existing problems and new challenges. In this paper, we present three areas of innovation in dCache: providing efficient access to data with NFS v4.1 pNFS, adoption of CDMI and WebDAV as an alternative to SRM for managing data, and integration with alternative authentication mechanisms.1. Introduction dCache[1] is a Free/Open-Source system that supports storing and retrieving large volumes of data by aggregating the capacity of many servers in a scalable fashion. It provides fast access to this data and scales to many petabytes of storage [2][3][4] [5]. dCache offers a number of advanced features like transparent access to tertiary storage systems (including several tape systems), file replication (for resilience and increased performance), hot-spot detection and mitigation and file integrity checking [6].As dCache is freely available, it is difficult to know exactly how many dCache instances are in use. For WLCG there are around 80 dCache clusters spread over some 60 sites throughout the world [7]. Although dCache is most heavily used in HEP [8][9], sites are increasingly offering dCache storage to non-HEP users [10][11]. This is possible because of dCache's strong emphasis on standards-compliance so that, in addition to HEP-specific protocols, dCache also supports the NFS v4.1/pNFS [12], HTTP/WebDAV [13] and FTP[5] protocols. This paper describes some of the recent changes in dCache that have been introduced to meet requirements of increasingly diverse dCache user-communities and how these changes are being adopted by HEP users. It will focus on three main areas: managing storage, data analysis and new ways of authentication. The remainder of the paper is structured to follow these topics with a section on CDMI and then by sections on NFS and authentication.

DUNE Offline Computing (Conceptual Design Report)

“…dCache -Several PB of dCache [130] storage are available for multiple purposes. dCache storage is not fully POSIX compliant but network file system (NFS) mounts with limited functionality are available on Fermilab interactive machines.…”

Integration of a heterogeneous compute resource in the ATLAS workflow

“…• Access to both grid-aware datasets on the distributed storage system dCache [14] and the local NEMO parallel filesystem BeeGFS [15].…”

Enabling Object Storage via shims for Grid Middleware

“…negotiated by a high level protocol (SRM) and a common Logical File Catalog technology for global replica management; to a brace of WAN protocols (HTTP, xrootd) with lighter negotiation, and a fragmentation of file catalogues across the WLCG experiments. However, the model for representation of storage at a given Storage Element has remained resolutely filesystem like, with SRM and all of the popular SE implementations (CASTOR [2], DPM [3], dCache [4]) exporting a common namespace with a hierarchical, directory-like, indexing of file names within it.…”