DUNE Production processing and workflow management software evaluation

Abstract

The Deep Underground Neutrino Experiment (DUNE) will be the world’s foremost neutrino detector when it begins taking data in the mid-2020s. Two prototype detectors, collectively known as ProtoDUNE, have begun taking data at CERN and have accumulated over 3 PB of raw and reconstructed data since September 2018. Particle interaction within liquid argon time projection chambers are challenging to reconstruct, and the collaboration has set up a dedicated Production Processing group to perform centralized reconstruction of the large ProtoDUNE datasets as well as to generate large-scale Monte Carlo simulation. Part of the production infrastructure includes workflow management software and monitoring tools that are necessary to efficiently submit and monitor the large and diverse set of jobs needed to meet the experiment’s goals. We will give a brief overview of DUNE and ProtoDUNE, describe the various types of jobs within the Production Processing group’s purview, and discuss the software and workflow management strategies are currently in place to meet existing demand. We will conclude with a description of our requirements in a workflow management software solution and our planned evaluation process.

Accelerating Machine Learning Inference with GPUs in ProtoDUNE Data Processing

“…Each production run during the beam period comprises several data files, each containing between 100 and 150 data records. In contrast to the previous work, in which DUNE simulation events were processed by submitting jobs locally to a dedicated queue, we submit jobs to process each file via the current standard DUNE workflow management and job submission systems [ 25 , 26 ], thus requiring no special treatment. Jobs may run either at Fermilab or one of several remote sites that we reach with opportunistic access enabled by the OSG Consortium [ 27 ].…”