Thrill: High-performance algorithmic distributed batch data processing with C++

Abstract

We present the design and a first performance evaluation of Thrill -a prototype of a general purpose big data processing framework with a convenient data-flow style programming interface. Thrill is somewhat similar to Apache Spark and Apache Flink with at least two main differences. First, Thrill is based on C++ which enables performance advantages due to direct native code compilation, a more cachefriendly memory layout, and explicit memory management. In particular, Thrill uses template meta-programming to compile chains of subsequent local operations into a single binary routine without intermediate buffering and with minimal indirections. Second, Thrill uses arrays rather than multisets as its primary data structure which enables additional operations like sorting, prefix sums, window scans, or combining corresponding fields of several arrays (zipping).We compare Thrill with Apache Spark and Apache Flink using five kernels from the HiBench suite. Thrill is consistently faster and often several times faster than the other frameworks. At the same time, the source codes have a similar level of simplicity and abstraction.

SIproc: an open-source biomedical data processing platform for large hyperspectral images

“…2). The use of asynchronous input/ output (IO) has been extensively studied for problems involving big data applications, particularly on distributed systems 23 such as supercomputers and clusters. Performance tuning in such cases involves selecting a number of parameters that are highly system dependent, particularly for heterogeneous computers.…”

The ISTI Rapid Response on Exploring Cloud Computing 2018

“…We explored using the Thrill [26] library to track the most energetic particles for the results of VPIC plasma physics simulation [32]. Thrill is a research project that aims to provide a bridge between big data analytics and HPC platforms.…”

DFG Priority Programme SPP 1736: Algorithms for Big Data

“…Another important step was to build a prototype of a tool for implementing algorithms that process large data sets on distributed memory machines. The result, Thrill [7], is based on C++, offers a rich set of operations on distributed arrays such as map, reduce, sort, merge, and prefix-sum. It can fuse pipelines of local operations into tight loops optimized at compile time, considerably outperforming established tools such as Spark or Flink.…”

“…: Broccoli [4] for semantic search, GENO 3 for generic optimization code generation, NetworKit [28] for network analysis, STXXL [11] for external-memory computing, and Thrill [7] for distributed batch data processing. The priority programme also creates visibility by its national and international events (e.g., Summer/Winter schools in Chennai 2016 and Tel Aviv 2017).…”