Rapid development of cloud-native intelligent data pipelines for scientific data streams using the HASTE Toolkit

Abstract

Background Large streamed datasets, characteristic of life science applications, are often resource-intensive to process, transport and store. We propose a pipeline model, a design pattern for scientific pipelines, where an incoming stream of scientific data is organized into a tiered or ordered “data hierarchy". We introduce the HASTE Toolkit, a proof-of-concept cloud-native software toolkit based on this pipeline model, to partition and prioritize data streams to optimize use of limited computing resources. Findings In our pipeline model, an “interestingness function” assigns an interestingness score to data objects in the stream, inducing a data hierarchy. From this score, a “policy” guides decisions on how to prioritize computational resource use for a given object. The HASTE Toolkit is a collection of tools to adopt this approach. We evaluate with 2 microscopy imaging case studies. The first is a high content screening experiment, where images are analyzed in an on-premise container cloud to prioritize storage and subsequent computation. The second considers edge processing of images for upload into the public cloud for real-time control of a transmission electron microscope. Conclusions Through our evaluation, we created smart data pipelines capable of effective use of storage, compute, and network resources, enabling more efficient data-intensive experiments. We note a beneficial separation between scientific concerns of data priority, and the implementation of this behaviour for different resources in different deployment contexts. The toolkit allows intelligent prioritization to be `bolted on' to new and existing systems – and is intended for use with a range of technologies in different deployment scenarios.

Simulation Intelligence: Towards a New Generation of Scientific Methods

“…See, for example, Simmhan et al [601] discuss pipeline designs for reliable, scalable data ingestion in a distributed environment in order to support the Pan-STARRS repository, one of the largest digital surveys that accumulates 100TB of data annually to support 300 astronomers. Blamey et al [602] also provide an informative walk through the challenges and implementations of cloud-native intelligent data pipelines for scientific data streams in life sciences. Deelman et al [603] advocate for co-located (or in situ) computational workflows, to minimize inefficiencies with distributed computing.…”

Data Management of Scientific Applications in a Reinforcement Learning-Based Hierarchical Storage System

“…Faster access to the most relevant data would significantly improve the efficiency of analysis. For example, HASTE project (Blamey et al, 2021) is a unique effort to address challenges related to scientific datasets. The HASTE project takes a hierarchical approach to acquisition, analysis, and interpretation of image data.…”