StoRM-GPFS-TSM: A new approach to hierarchical storage management for the LHC experiments

Cavalli, Alessandro; dell’Agnello, Luca; Ghiselli, A.; Gregori, Daniele; Magnoni, L.; Martelli, Barbara; Mazzucato, M.; Prosperini, Andrea; Ricci, P.; Ronchieri, Elisabetta; Sapunenko, V.; Vagnoni, V.; Vitlacil, Dejan; Zappi, Riccardo

doi:10.1088/1742-6596/219/7/072030

Cited by 10 publications

(6 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The farm worker nodes access the storage through a standard 10 Gb Ethernet network using the posix file protocol, and access the ATLAS experiment software via the CernVM File System (CVMFS) [7]. The IBM Tivoli Storage Manager (TSM) [8] system is used to manage data access on tape via the GEMMS interface [9], a complete Grid-enabled Hierarchical Storage Management (HSM) solution based on StoRM/GPFS and TSM, developed and deployed at CNAF.…”

Section: The Tier-1 Sitementioning

confidence: 99%

ATLAS computing activities and developments in the Italian Grid cloud

Rinaldi

Annovi

Antonelli

et al. 2012

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

The large amount of data produced by the ATLAS experiment needs new computing paradigms for data processing and analysis, which involve many computing centres spread around the world. The computing workload is managed by regional federations, called "clouds". The Italian cloud consists of a main (Tier-1) center, located in Bologna, four secondary (Tier-2) centers, and a few smaller (Tier-3) sites. In this contribution we describe the Italian cloud facilities and the activities of data processing, analysis, simulation and software development performed within the cloud, and we discuss the tests of the new computing technologies contributing to evolution of the ATLAS Computing Model.

show abstract

Section: The Tier-1 Sitementioning

confidence: 99%

ATLAS computing activities and developments in the Italian Grid cloud

Rinaldi

Annovi

Antonelli

et al. 2012

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

show abstract

“…Storage technologies covered in these tests were: Castor [20],dCache[21], DPM[22], EOS [23], Hadoop/BeStMan [24], Lustre[25], L-Store [26], StoRM/GPFS [27], xrootd [28]. The campaign was very successful.…”

Section: Components Of a Common Solution 51 Information Providermentioning

confidence: 99%

Data Storage Accounting and Verification at LHC experiments

Huang

Lanciotti²,

Magini³

et al. 2012

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

All major experiments at the Large Hadron Collider (LHC) need to measure real storage usage at the Grid sites. This information is equally important for resource management, planning, and operations. To verify the consistency of central catalogs, experiments are asking sites to provide a full list of the files they have on storage, including size, checksum, and other file attributes. Such storage dumps, provided at regular intervals, give a realistic view of the storage resource usage by the experiments. Regular monitoring of the space usage and data verification serve as additional internal checks of the system integrity and performance. Both the importance and the complexity of these tasks increase with the constant growth of the total data volumes during the active data taking period at the LHC. The use of common solutions helps to reduce the maintenance costs, both at the large Tier1 facilities supporting multiple virtual organizations and at the small sites that often lack manpower. We discuss requirements and solutions to the common tasks of data storage accounting and verification, and present experiment-specific strategies and implementations used within the LHC experiments according to their computing models.

show abstract

“…For otherwise cold-storage settings where data requests are few and far between, magnetic tapes are the method of choice in industry (Frank et al 2012, Hope 2019. Examples include high-resolution digital content for media and entertainment platforms (Coughlin 2019); deep-ocean seismic data for the oil industry (Gaul et al 2007); modeling, simulation, and analysis of space missions (NASA 2021, Albani et al 2020; and multi-year research data for projects such as the Large Hadron Collider (Cavalli et al 2010).…”

Section: Introductionmentioning

confidence: 99%

On Approximate Sequencing Policies for Linear Storage Devices

Cardonha¹,

Ciré²,

Real³

2021

Preprint

View full text Add to dashboard Cite

This paper investigates scheduling policies for file retrieval in linear storage devices, such as magnetic tapes. Tapes are the technology of choice for long-term storage in data centers due to their low cost per capacity, reliability, and data security. While scheduling problems associated with data retrieval in tapes are classical, existing works focus on more straightforward heuristic approaches due to limited computational times imposed by standard tape specifications. Our first contribution is a theoretical investigation of three standard policies, presenting their worstcase performance and special cases of practical relevance for which they are optimal. Next, we show that the problem is polynomially solvable via two interleaved recursive models, albeit with high computational complexity. We leverage our previous results to develop two new scheduling policies with constant-ratio performance and low computational cost. Finally, we investigate properties associated with the online variant of the problem, presenting a new constant-factor competitive algorithm. Our numerical analysis on synthetic and real-world tapes from an industry partner provides insights into dataset configurations where each policy is more effective, which is of relevance to data center managers. In particular, our new best-performing policy is practical for large datasets and significantly improves upon standard algorithms in the area.

show abstract

StoRM-GPFS-TSM: A new approach to hierarchical storage management for the LHC experiments

Cited by 10 publications

References 3 publications

ATLAS computing activities and developments in the Italian Grid cloud

ATLAS computing activities and developments in the Italian Grid cloud

Data Storage Accounting and Verification at LHC experiments

On Approximate Sequencing Policies for Linear Storage Devices

Contact Info

Product

Resources

About