CERNBox + EOS: end-user storage for science

Mascetti, Luca; Labrador, Hugo González; Lamanna, M.; Mościcki, JT; Peters, AJ

doi:10.1088/1742-6596/664/6/062037

Cited by 26 publications

(22 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Inside the container, a Jupyter server process is created to handle the requests of the users when they are working in the notebook interface. In addition, the container has access to CVMFS [9] for software which include Apache Spark Software and CERNBox [10] for data from their sessions.…”

Section: Spark Connectormentioning

confidence: 99%

Apache Spark usage and deployment models for scientific computing

et al. 2019

View full text Add to dashboard Cite

This talk is about sharing our recent experiences in providing data analytics platform based on Apache Spark for High Energy Physics, CERN accelerator logging system and infrastructure monitoring. The Hadoop Service has started to expand its user base for researchers who want to perform analysis with big data technologies. Among many frameworks, Apache Spark is currently getting the most traction from various user communities and new ways to deploy Spark such as Apache Mesos or Spark on Kubernetes have started to evolve rapidly. Meanwhile, notebook web applications such as Jupyter offer the ability to perform interactive data analytics and visualizations without the need to install additional software. CERN already provides a web platform, called SWAN (Service for Web-based ANalysis), where users can write and run their analyses in the form of notebooks, seamlessly accessing the data and software they need. The first part of the presentation talks about several recent integrations and optimizations to the Apache Spark computing platform to enable HEP data processing and CERN accelerator logging system analytics. The optimizations and integrations, include, but not limited to, access of kerberized resources, xrootd connector enabling remote access to EOS storage and integration with SWAN for interactive data analysis, thus forming a truly Unified Analytics Platform. The second part of the talk touches upon the evolution of the Apache Spark data analytics platform, particularly sharing the recent work done to run Spark on Kubernetes on the virtualized and container-based infrastructure in Openstack. This deployment model allows for elastic scaling of data analytics workloads enabling efficient, on-demand utilization of resources in private or public clouds.

show abstract

Section: Spark Connectormentioning

confidence: 99%

Apache Spark usage and deployment models for scientific computing

et al. 2019

View full text Add to dashboard Cite

show abstract

“…On the other hand, CERN makes available to every user a private and synchronised portion of EOS, called CERNBox [29]. Section 4 is dedicated to the description of this type of storage.…”

Section: Storagementioning

confidence: 99%

SWAN: A service for interactive analysis in the cloud

Piparo

Tejedor

Mató

et al. 2018

Future Generation Computer Systems

View full text Add to dashboard Cite

SWAN (Service for Web based ANalysis) is a platform to perform interactive data analysis in the cloud. SWAN allows users to write and run their data analyses with only a web browser, leveraging on the widely-adopted Jupyter notebook interface. The user code, executions and data live entirely in the cloud. SWAN makes it easier to produce and share results and scientific code, access scientific software, produce tutorials and demonstrations as well as preserve analyses. Furthermore, it is also a powerful tool for non-scientific data analytics.This paper describes how a pilot of the SWAN service was implemented and deployed at CERN. Its backend combines state-of-the-art software technologies with a set of existing IT services such as user authentication, virtual computing infrastructure, mass storage, file synchronisation and sharing, specialised clusters and batch systems.The added value of this combination of services is discussed, with special focus on the opportunities offered by the CERNBox service and its massive storage backend, EOS. In particular, it is described how a cloud-based analysis model benefits from synchronised storage and sharing capabilities.

show abstract

“…A stand-alone test was performed using a dedicated test cluster of 3 nodes (1 name node, 2 data nodes) each with an Intel(R) Xeon(R) CPU E5-2650 @ 2.00GHz with 32 cores, 128 GB RAM and 10 Gb/s network connection. Files in different formats (text, parquet [15], ROOT) were read either from the public or CERNBOX [16] EOS instances at CERN, where the public EOS instance shows decreased performance characteristics because of its physical size and access pattern optimization compared to the CERNBOX instance. The results are shown in Table 1.…”

Section: The Spark Analysis Workflowmentioning

confidence: 99%

CMS Analysis and Data Reduction with Apache Spark

Gutsche

Canali

Cremer³

et al. 2018

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

Experimental Particle Physics has been at the forefront of analyzing the world's largest datasets for decades. The HEP community was among the first to develop suitable software and computing tools for this task. In recent times, new toolkits and systems for distributed data processing, collectively called "Big Data" technologies have emerged from industry and open source projects to support the analysis of Petabyte and Exabyte datasets in industry. While the principles of data analysis in HEP have not changed (filtering and transforming experiment-specific data formats), these new technologies use different approaches and tools, promising a fresh look at analysis of very large datasets that could potentially reduce the time-to-physics with increased interactivity. Moreover these new tools are typically actively developed by large communities, often profiting of industry resources, and under open source licensing. These factors result in a boost for adoption and maturity of the tools and for the communities supporting them, at the same time helping in reducing the cost of ownership for the end-users. In this talk, we are presenting studies of using Apache Spark for end user data analysis. We are studying the HEP analysis workflow separated into two thrusts: the reduction of centrally produced experiment datasets and the end-analysis up to the publication plot. Studying the first thrust, CMS is working together with CERN openlab and Intel on the CMS Big Data Reduction Facility. The goal is to reduce 1 PB of official CMS data to 1 TB of ntuple output for analysis. We are presenting the progress of this 2-year project with first results of scaling up Spark-based HEP analysis. Studying the second thrust, we are presenting studies on using Apache Spark for a CMS Dark Matter physics search, investigating Spark's feasibility, usability and performance compared to the traditional ROOT-based analysis.

show abstract

CERNBox + EOS: end-user storage for science

Cited by 26 publications

References 2 publications

Apache Spark usage and deployment models for scientific computing

Apache Spark usage and deployment models for scientific computing

SWAN: A service for interactive analysis in the cloud

CMS Analysis and Data Reduction with Apache Spark

Contact Info

Product

Resources

About