Wrangling distributed computing for high-throughput environmental science: An introduction to HTCondor

Abstract

Biologists and environmental scientists now routinely solve computational problems that were unimaginable a generation ago. Examples include processing geospatial data, analyzing -omics data, and running large-scale simulations. Conventional desktop computing cannot handle these tasks when they are large, and high-performance computing is not always available nor the most appropriate solution for all computationally intense problems. High-throughput computing (HTC) is one method for handling computationally intense research. In contrast to high-performance computing, which uses a single "supercomputer," HTC can distribute tasks over many computers (e.g., idle desktop computers, dedicated servers, or cloud-based resources). HTC facilities exist at many academic and government institutes and are relatively easy to create from commodity hardware. Additionally, consortia such as Open Science Grid facilitate HTC, and commercial entities sell cloud-based solutions for researchers who lack HTC at their institution. We provide an introduction to HTC for biologists and environmental scientists. Our examples from biology and the environmental sciences use HTCondor, an open source HTC system.

Network inference with Granger causality ensembles on single-cell transcriptomics

“…The entire SINGE workflow for all hyperparameters and replicates requires 1,219.4 h. However, Jump3 and SINGE are highly parallelizable. We deployed them on our local high-throughput computing cluster using HTCondor (Erickson et al, 2018), which connects to the OSG (Pordes et al, 2007). In this highthroughput setting, we can run the entire SINGE algorithm in 36 h and the Jump3 algorithm in 72 h. SINGE can also be configured to run on a single workstation with appropriate changes to the hyperparameters.…”

Paths to computational fluency for natural resource educators, researchers, and managers

“…HPC works best with large computer tasks that are often broken down into tightly linked smaller tasks. In contrast, HTC works best with large computer tasks that can be broken down into many smaller tasks and run independently (Erickson et al, 2018). Historically, HPC required large, onsite computers, such as the super‐computers, found on major research universities or national laboratories.…”

A new method for geomorphological studies and land cover classification using Machine Learning techniques

“…High Throughput Computing (HTC) represents a method for handling this computationally intense task, which let the users split a big task into many small independent ones (a.k.a. jobs) that are distributed across a computer cluster [9].…”