Faster Coroutine Pipelines: A Reconstruction

Abstract

Spivey has recently presented a novel functional representation that supports the efficient composition, or merging, of coroutine pipelines for processing streams of data. This representation was inspired by Shivers and Might's three-continuation approach and is shown to be equivalent to a simple yet inefficient executable specification. Unfortunately, neither Shivers and Might's original work nor the equivalence proof sheds much light on the underlying principles allowing the derivation of this efficient representation from its specification. This paper gives the missing insight by reconstructing a systematic derivation in terms of known transformation steps from the simple specification to the efficient representation. This derivation sheds light on the limitations of the representation and on its applicability to other settings. In particular, it has enabled us to obtain a similar representation for pipes featuring two-way communication, similar to the Haskell pipes library. Our benchmarks confirm that this two-way representation retains the same improved performance characteristics.

Faster coroutine pipelines: A reconstruction

“…The continuation-based encoding described in this paper has been made available as a library on github (Pieters, 2018b). helpermr −1 :: ProdPar x (Producer x) → Producer x helpermr −1 (ProdPar f ) = f (ConsPar (λx prod → Producer x (helpermr −1 prod)))…”

“…The continuation-based encoding described in this paper has been made available as a library on github (Pieters, 2018b).…”

“…All benchmarks are executed using the criterion library (O'Sullivan, 2009) on an Intel Core i7-6600U at 2.60 GHz with 8 GB memory running Ubuntu 16.04 and GHC 8.8.3, with -O2 enabled. In the following sections, we highlight the relevant parts of the benchmarking code, but we refer to the full implementation for the details (Pieters, 2018a). The library versions used are pipes v4.3.13, conduit v1.3.2, and streamly v0.7.1.…”

“…Section 7 discusses related work and Section 8 concludes this paper. This paper is based on an earlier publication: "Faster Coroutine Pipelines: A Reconstruction" (Pieters & Schrijvers, 2019). The main changes are the addition of examples and extended explanations in Sections 3 and 4, an overhaul of Section 5 with more extensive explanation and examples, and in Section 6 where an additional stream processing library streamly is added to the benchmarks, more information regarding the benchmarks is added, we investigate some additional use case-driven benchmarks and we discuss several performance optimization techniques applied on the continuation-based encoding.…”