Monitoring WLCG with lambda-architecture: a new scalable data store and analytics platform for monitoring at petabyte scale.

An efficient strategy for the collection and storage of large volumes of data for computation

“…The performance test results show that eliminating any transformation at the data ingestion level and moving it to the analytics job is beneficial as the overall process time is reduced, untempered raw data are kept in the storage level for fault-tolerance, and the required transformation can be done as and when required using a framework such as MapReduce. The presented results show that this proposed approach outperformed the approach employed at the WLCG and following this work the new approach has been adopted by the WLCG and it has been used for collecting, storing, and analysing metadata at CERN since April 2015 [6]. This approach can be easily applied to other use cases (e.g.…”

“…This requires the gathering of metadata which describes the data (e.g. transfer time) from geographically distributed sources and the processing of such information to extract the relevant information for the WLCG group [6]. Since the LHC experiments are so well known and many studies have been conducted on their demands and requirements, one can use the LHC experiments as a suitable case study for this research.…”

Optimised Lambda Architecture for Monitoring Scientific Infrastructure

“…Old backlog injection is therefore very common when a long running job is completed. It is necessary to have both a batch layer as well as a streaming layer (real-time) as presented in [3]. However, this requires better mechanisms in place to simplify the system.…”

“…Due to the high volume and velocity of the events that are produced, the traditional methods are not optimal. The WLCG Data acTivities (WDT) dashboards are a set of monitoring tools utilised to monitor data access, and transfer data across WLCG sites through various protocols and service [3]. The monitored services using WDT include the HTTP federations, XRootD, ATLAS Distributed Data Management (DDM) system, and the File Transfer Service (FTS).…”

“…The monitored services using WDT include the HTTP federations, XRootD, ATLAS Distributed Data Management (DDM) system, and the File Transfer Service (FTS). The WDT use case is one of the most data intensive applications, and the WDT dashboards struggle from the restriction of the original processing infrastructure [3]. The original architecture relies on an Oracle database to store, process, and serve the monitoring data [3].…”