The IceProd framework: Distributed data processing for the IceCube neutrino observatory

Abstract

IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, identify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. This paper presents the first detailed description of IceProd, a lightweight distributed management system designed to meet these requirements. It is driven by a central database in order to manage mass production of simulations and analysis of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, HTCondor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework.

IceProd - a dataset management system for IceCube: update

“…It consists of 5160 optical sensors buried between 1450 and 2450 meters below the surface of the South Pole ice sheet and is designed to detect interactions of neutrinos of astrophysical origin. The IceProd [2] software framework was developed by IceCube in 2006 to manage distributed simulation workloads, primarily for data provenance and dataset submission to HTCondor [3]. It has been used in production for 12 years to handle the data processing and simulation needs for the IceCube collaboration.…”

IceCube simulation production and the transition to IceProd2

“…IceProd has served the IceCube collaboration well over the last 8 years, processing Monte Carlo simulations, detector data, and analysis levels [1]. It has run thousands of CPU-core years and stored over 2PB of data.…”

Scalable generation of random graphs

“…The parallel generators discussed so far may emit multi-edges. 1 The distributed memory generator by Alam et al [12] (see Section 2.4.2.2) produces simple BA networks as a special case of the Node Copy model.…”

“…of communities follows endogenously and is bounded by C=O(n) even if nodes are members in ν=O(1) communities. 2 Depending on the mixing parameter 0 < µ < 1 In the case of overlapping communities, the internal degree is evenly split among all communities of the node. Both the computation of d in i and the division d in i /ν i into several communities use non-deterministic rounding to avoid biases.…”

“…• The expected number of edges computed for high degree vertices (as well as the number of vertices recomputed during the outward search) can pessimistically be bounded by assuming that vertices in the inner annuli (see Lemma 7.15) all have maximum degree ∆ = n 1/(2α)+o (1) . [159] This yields an expected number of O (n 1−α (Pd) α n 1/(2α) ) edges.…”