Runtime resource allocation for software pipelines

Abstract

Systems continue to comprise a rapidly growing number of cores on a single chip to gain performance benefits from parallel processing. A key challenge is how their computational resources can be used efficiently, which depends to a large degree on how their resources are allocated to the applications. In this paper, we describe our current research for addressing this challenge and highlight current and upcoming hurdles that need to be addressed.

Protein allocation and utilization in the versatile chemolithoautotroph Cupriavidus necator

“…The openMS/KNIME workflow for MS data processing is available at https://github.com/m-jahn/openMS-workflows , (copy archived at swh:1:rev:dd4e0a20a39300cac9ad89840862348895e9f907 , Jahn, 2021c ). The revised genome scale model of C. necator H16 is available at https://github.com/m-jahn/genome-scale-models , (copy archived at swh:1:rev:d2cdcdfbdf140694993a3108b1a10715566f09aa , Jahn, 2021b ).…”

“…The openMS/KNIME workflow for MS data processing is available at https://github.com/m-jahn/openMS-workflows , (copy archived at swh:1:rev:dd4e0a20a39300cac9ad89840862348895e9f907 , Jahn, 2021c ). The revised genome scale model of C. necator H16 is available at https://github.com/m-jahn/genome-scale-models , (copy archived at swh:1:rev:d2cdcdfbdf140694993a3108b1a10715566f09aa , Jahn, 2021b ). The resource balance analysis (RBA) model of C. necator H16 is available at https://github.com/m-jahn/Bacterial-RBA-models , (copy archived at swh:1:rev:efe12f7d53810e1aec618b2b2da0fa8a49aec1c5 , Jahn, 2021a ).…”

“…The resource balance analysis (RBA) model of C. necator H16 is available at https://github.com/m-jahn/Bacterial-RBA-models , (copy archived at swh:1:rev:efe12f7d53810e1aec618b2b2da0fa8a49aec1c5 , Jahn, 2021a ). The code used to process TnSeq data from raw fastq files (read trimming, filtering, mapping to genome) is available at https://github.com/m-jahn/TnSeq-pipe , (copy archived at swh:1:rev:1f256a366f772a2450c2bcfecc43bb2181efc989 , Jahn, 2021e ). The code used to process BarSeq data from raw fastq files is available at https://github.com/Asplund-Samuelsson/rebar , (copy archived at swh:1:rev:5a4dbad30041bc0510a9a2eed55b1a47f705ff51 , Asplund-Samuelsson, 2021 ).…”

Pipel: exploiting resource reorganisation to optimise performance of pipeline-structured applications in the cloud

“…The author created a definition, design and implementation of a new framework to automatically provision the virtual resources needed to execute a multi-tier application to meet the user performance requirements and to minimise the number of provisioned VMs. Jahn et al (2015) present an optimal and centralised method to allocate resources to software-pipelined applications. To diminish the negative effects of unreliable processing units, their study shows how self-organisation can effectively restore the integrity of such a hierarchy when it is corrupted by a failing core.…”

“…Pipeline-structured applications can easily become time-consuming, since performance depends on the number of input tasks, features of each task, the number of stages and the processing behaviour of each stage (Rodriguez and Buyya, 2015;Jahn et al, 2015). A traditional problem in pipeline applications occurs when a stage becomes slower than others, so being the system bottleneck (Malawski et al, 2012;Beltran, 2015).…”