Streamlining CASTOR to manage the LHC data torrent

Presti, Giuseppe Lo; Curull, X. Espinal; Cano, E.; Fiorini, Barbara; Ieri, A; Murray, S.; Ponce, S.; Sindrilaru, Elvin Alin

doi:10.1088/1742-6596/513/4/042031

Cited by 4 publications

(2 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[6] The initial approach at RAL was to simply reproduce the CERN infrastructure, on the grounds that it was known to work and seemed like a definite upgrade to the DLF. However, this strategy proved to be much more complex to implement that originally expected -the system was designed to be built on top of CERN's Agile Infrastructure Platform [7], which STFC has no analogue to.…”

Section: The Cern Hadoop Solutionmentioning

confidence: 99%

Using the ELK Stack for CASTOR Application Logging at RAL

Appleyard¹,

Adams²

2016

Proceedings of International Symposium on Grids and Clouds 2015 — PoS(ISGC2015)

View full text Add to dashboard Cite

Logging of application and system information is crucial to running any service. Whether it is looking for trends in use, investigating specific problems or analysing security incidents, logs are an essential source of information for any service manager or system administrator. However, distributed applications produce large volumes of logging information from a number of disparate, but related, daemons running on an increasing number of machines. In this scenario, log analysis with traditional command line tools such as awk and grep is hugely inefficient. In addition, presenting the results of analysis using tools like spreadsheets is a time consuming process for a busy administrator. At STFC, the Scientific Computing Department has adopted the ELK stack (www.elastic.co) for collating of logs from the mass storage system. The ELK stack is built from three open source components -Elasticsearch, Logstash, and Kibana. It is currently gathering information from about 300 machines at the Rutherford Appleton Laboratory covering both the WLCG Tier 1 and local facilities storage systems. In this presentation, we discuss the evolution of Elasticsearch within STFC and present specific use cases where it has clear advantages over traditional methods. We also compare it with the previous logging system developed at CERN (DLF) and show that the ELK stack is a more generically useful toolset. Finally, we demonstrate its usefulness for looking at both trends and specific event analysis.

show abstract

Section: The Cern Hadoop Solutionmentioning

confidence: 99%

Using the ELK Stack for CASTOR Application Logging at RAL

Appleyard¹,

Adams²

2016

Proceedings of International Symposium on Grids and Clouds 2015 — PoS(ISGC2015)

View full text Add to dashboard Cite

show abstract

“…negotiated by a high level protocol (SRM) and a common Logical File Catalog technology for global replica management; to a brace of WAN protocols (HTTP, xrootd) with lighter negotiation, and a fragmentation of file catalogues across the WLCG experiments. However, the model for representation of storage at a given Storage Element has remained resolutely filesystem like, with SRM and all of the popular SE implementations (CASTOR [2], DPM [3], dCache [4]) exporting a common namespace with a hierarchical, directory-like, indexing of file names within it.…”

Section: Introductionmentioning

confidence: 99%

Enabling Object Storage via shims for Grid Middleware

Skipsey,

De Witt,

Dewhurst

et al. 2015

Preprint

View full text Add to dashboard Cite

The Object Store model has quickly become the basis of most commercially successful mass storage infrastructure, backing so-called "Cloud" storage such as Amazon S3, but also underlying the implementation of most parallel distributed storage systems. Many of the assumptions in Object Store design are similar, but not identical, to concepts in the design of Grid Storage Elements, although the requirement for "POSIX-like" filesystem structures on top of SEs makes the disjunction seem larger. As modern Object Stores provide many features that most Grid SEs do not (block level striping, parallel access, automatic file repair, etc.), it is of interest to see how easily we can provide interfaces to typical Object Stores via plugins and shims for Grid tools, and how well experiments can adapt their data models to them. We present evaluation of, and first-deployment experiences with, (for example) Xrootd-Ceph interfaces for direct object-store access, as part of an initiative within GridPP[1] hosted at RAL. Additionally, we discuss the tradeoffs and experience of developing plugins for the currently-popular Ceph parallel distributed filesystem for the GFAL2 access layer, at Glasgow.

show abstract