Accessing opportunistic resources with Bosco

Weitzel, Derek; Sfiligoi, I.; Bockelman, Brian; Frey, Jaime; Wuerthwein, Frank; Fraser, Dan; Swanson, David

doi:10.1088/1742-6596/513/3/032105

Cited by 14 publications

(7 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The HPC cluster execution environment is made up of a remote compute cluster with interfaces for job control and data management. Common interfaces include Globus GRAM/GridFTP, CreamCE, and Bosco [38] if only SSH access is provided to the cluster. The cluster is usually configured for parallel job execution.…”

Section: Hpc Cluster Environmentmentioning

confidence: 99%

Pegasus, a workflow management system for science automation

Deelman

Vahi

Juve

et al. 2015

Future Generation Computer Systems

696

402

View full text Add to dashboard Cite

Modern science often requires the execution of large-scale, multi-stage simulation and data analysis pipelines to enable the study of complex systems. The amount of computation and data involved in these pipelines requires scalable workflow management systems that are able to reliably and efficiently coordinate and automate data movement and task execution on distributed computational resources: campus clusters, national cyberinfrastructures, and commercial and academic clouds. This paper describes the design, development and evolution of the Pegasus Workflow Management System, which maps abstract workflow descriptions onto distributed computing infrastructures. Pegasus has been used for more than twelve years by scientists in a wide variety of domains, including astronomy, seismology, bioinformatics, physics and others. This paper provides an integrated view of the Pegasus system, showing its capabilities that have been developed over time in response to application needs and to the evolution of the scientific computing platforms. The paper describes how Pegasus achieves reliable, scalable workflow execution across a wide variety of computing infrastructures.

show abstract

Section: Hpc Cluster Environmentmentioning

confidence: 99%

Pegasus, a workflow management system for science automation

Deelman

Vahi

Juve

et al. 2015

Future Generation Computer Systems

696

402

View full text Add to dashboard Cite

show abstract

“…In 2016 we switched our efforts to the Edison and Cori clusters and in 2017 we started provisioning resources on Cori and Edison through HEPCloud. NERSC is integrated into the HEPCloud resource provisioning via GlideInWMS [4], HT-Condor [5] [6] and Bosco [7]. Bosco allows the HEPCloud pilot factory to submit pilot jobs to both Cori and Edison via a remote ssh tunnel.…”

Section: Nersc Cori and Edisonmentioning

confidence: 99%

HPC resource integration into CMS Computing via HEPCloud

et al. 2019

View full text Add to dashboard Cite

The higher energy and luminosity from the LHC in Run 2 have put increased pressure on CMS computing resources. Extrapolating to even higher luminosities (and thus higher event complexities and trigger rates) beyond Run 3, it becomes clear that simply scaling up the the current model of CMS computing alone will become economically unfeasible. High Performance Computing (HPC) facilities, widely used in scientific computing outside of HEP, have the potential to help fill the gap. Here we describe the U.S.CMS efforts to integrate US HPC resources into CMS Computing via the HEPCloud project at Fermilab. We present advancements in our ability to use NERSC resources at scale and efforts to integrate other HPC sites as well. We present experience in the elastic use of HPC resources, quickly scaling up use when so required by CMS workflows. We also present performance studies of the CMS multi-threaded framework on both Haswell and KNL HPC resources. *

show abstract

“…The OSG Hosted CE service is an HTCondor-CE gateway working over SSH. Pilot Jobs are submitted to the remote cluster batch system through an external host running HTCondor-CE and submitting via Bosco [9], as shown in Figure 3.…”

Section: The Osg Hosted Ce Servicementioning

confidence: 99%

Deploying and extending CMS Tier 3s using VC3 and the OSG Hosted CE service

et al. 2019

View full text Add to dashboard Cite

CMS Tier 3 centers, frequently located at universities, play an important role in the physics analysis of CMS data. Although different computing resources are often available at universities, meeting all requirements to deploy a valid Tier 3 able to run CMS workflows can be challenging in certain scenarios. For instance, providing the right operating system (OS)with access to the CERNVM File System (CVMFS) on the worker nodes or having a Compute Element (CE) on the submit host is not always allowed or possible due to e.g: lack of root access to the nodes, TCP port network policies, maintenance of a C,etc. The Notre Dame group operates a CMS Tier 3 with 1K cores. In addition to this, researchers have access to an opportunistic pool with +25K cores that are used via lobster for CMS jobs, but cannot be used with other standard CMS submission tools on the grid like CRAB, as these resources are not part of the Tier 3 due to its opportunistic nature. This work describes the use of VC3, a service for automating the deployment of virtual cluster infrastructures, in order to provide the environment (user-space CVMFS access and customized OS via singularity containers) needed for CMS workflows to work. Also, we describe its integration with the OSG Hosted CE service, to add these resources to CMS as part of our existing Tier 3 in a seamless way.

show abstract

Accessing opportunistic resources with Bosco

Cited by 14 publications

References 6 publications

Pegasus, a workflow management system for science automation

Pegasus, a workflow management system for science automation

HPC resource integration into CMS Computing via HEPCloud

Deploying and extending CMS Tier 3s using VC3 and the OSG Hosted CE service

Contact Info

Product

Resources

About