HarmonicIO: Scalable Data Stream Processing for Scientific Datasets

Torruangwatthana, Preechakorn; Wieslander, Håkan; Blamey, Ben; Hellander, Andreas; Toor, Salman

doi:10.1109/cloud.2018.00126

Cited by 9 publications

(4 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To address this gap, we have previously developed and reported on a stream processing framework focusing on scienti c computing applications, HarmonicIO [15]. HarmonicIO sacri ces some of these features, and is intended for lowerfrequency applications (towards kHz, not MHz), and was able to achieve better streaming performance under some conditions in one study for larger message sizes [14].…”

Section: Background: Stream Processing and Work Ow Managementmentioning

confidence: 99%

Rapid development of cloud-native intelligent data pipelines for scientific data streams using the HASTE Toolkit

Blamey

Toor

Dahlö

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

This paper introduces the HASTE Toolkit, a cloud-native software toolkit capable of partitioning data streams in order to prioritize usage of limited resources. This in turn enables more efficient data-intensive experiments. We propose a model that introduces automated, autonomous decision making in data pipelines, such that a stream of data can be partitioned into a tiered or ordered data hierarchy. Importantly, the partitioning is online and based on data content rather than a priori metadata. At the core of the model are interestingness functions and policies. Interestingness functions assign a quantitative measure of interestingness to a single data object in the stream, an interestingness score. Based on this score, a policy guides decisions on how to prioritize computational resource usage for a given object. The HASTE Toolkit is a collection of tools to adapt data stream processing to this pipeline model. The result is smart data pipelines capable of effective or even optimal use of e.g. storage, compute and network bandwidth, to support experiments involving rapid processing of scientific data characterized by large individual data object sizes. We demonstrate the proposed model and our toolkit through two microscopy imaging case studies, each with their own interestingness functions, policies, and data hierarchies. The first deals with a high content screening experiment, where images are analyzed in an on-premise container cloud with the goal of prioritizing the images for storage and subsequent computation. The second considers edge processing of images for upload into the public cloud for a real-time control loop for a transmission electron microscope.

show abstract

Section: Background: Stream Processing and Work Ow Managementmentioning

confidence: 99%

Rapid development of cloud-native intelligent data pipelines for scientific data streams using the HASTE Toolkit

Blamey

Toor

Dahlö

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…One of the components of the HASTE platform is the stream processing framework HarmonicIO (HIO), introduced in [6]. HIO uses Docker containers as processing engines (PEs) to process the streamed messages; these are designed and provided by the client based on a template.…”

Section: The Harmonicio Streaming Frameworkmentioning

confidence: 99%

“…The most popular self-contained application design scheme is the containerization approach. Based on Docker 4 , LXD 5 or Singularity 6 container software, applications can easily be deployed on a variety of resources.…”

Section: Introductionmentioning

confidence: 99%

“…Motivated by applications in large-scale processing of microscopy images, we recently developed HarmonicIO, a data streaming framework geared towards efficient processing of streams composed of large individual objects [6]. Recent performance benchmark comparisons of HarmonicIO with the Spark and Kafka streaming frameworks illustrate the increased throughput that can be expected from HarmonicIO in that scenario [7].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation