StoRM: A flexible solution for Storage Resource Manager in grid

Abstract

In these times, scientific data intensive applications generate ever-increasing volumes of data that need to be stored, managed, and shared between geographically distributed communities. Data centres are normally able to provide tens of petabytes of storage space through a large variety of heterogeneous storage and file systems. However, storage systems shared by applications need a common data access mechanism, which allocates storage space dynamically, manages stored content, and automatically remove unused data to avoid clogging data stores. To accommodate these needs, the concept of Storage Resource Managers (SRMs) was devised in the context of a project that involved High Energy Physics (HEP) and Nuclear Physics (NP). The Storage Resource Manager (SRM) interface specification was defined and evolved into an international collaboration in the context of the Open Grid Forum (OGF). SRM interface provides the technology needed to share geographically distributed heterogeneous storage resources, with an effective and common interface regardless of the type of the back-end system being used. By implementing the SRM interface, grid storage services provide a consistent homogeneous interface to the Grid to manage storage resource as well as advanced functionality such as dynamic space allocation and file management on shared storage systems. Within Worldwide LHC Grid project exists more than five interoperating implementations of SRM services, and every one shows peculiarity. In this paper, we describe the flexibility of StoRM service, an implementation of the Storage Resource Manager Interface version 2.2. StoRM is designed to foster the adoption of cluster file systems, and thanks to the marked flexibility, StoRM can be used in small data centre with human resource deficiency to administer an other grid service and, at the same time, capable to grow in terms of storage managed and workload. StoRM can be used to manage any storage resources with any kind of POSIX filesystem in a transparent way. As demonstration of the StoRM flexibility, the paper describes how applications scheduled via Grid can access files on a file-system directly via POSIX calls, how StoRM can be deployed in a clustered configuration to address scalability needs and finally how StoRM can be used to manage also storage classes based on Storage Cloud, like Amazon Simple Storage Service (S3).

Large Scale Data Handling experience at INFN-CNAF Data Center

“…For what concerns data management, CNAF is a StoRM [2] site. We run a dedicated StoRM endpoint for each major experiment, plus two endpoints which are shared among the many others.…”

Towards Predictive Maintenance with Machine Learning at the INFN-CNAF computing centre

“…CNAF has been involved in the development of a large set of software products. In the area of middleware for distributed systems CNAF supports, maintains and further evolves widely-used products, such as StoRM [5].…”

Collection and harmonization of system logs and prototypal Analytics services with the Elastic (ELK) suite at the INFN-CNAF computing centre

“…Moreover, high performance disk-storage solutions have become increasingly important to deal with the large I/O throughput required by the High Energy Physics (HEP) community for both data analysis and Monte Carlo simulations. Given such requirements, the HEP Grid community has developed and implemented the Storage Resource Manager (SRM) interface [12]. From one side a client can request the SRM to reserve space and manage files and directories; from the other side, SRM is able to dynamically decide which files to keep inside the storage space and which to remove e.g.…”