$$\textsf {CircuitFlow}$$: A Domain Specific Language for Dataflow Programming

Abstract

Dataflow applications, such as machine learning algorithms, can run for days, making it desirable to have assurances that they will work correctly. Current tools are not good enough: too often the interactions between tasks are not type-safe, leading to undesirable runtime errors. This paper presents a new declarative Haskell Embedded DSL (eDSL) for dataflow programming: CircuitFlow. Defined as a Symmetric Monoidal Preorder (SMP) on data that models dependencies in the workflow, it has a strong mathematical basis, refocusing on how data flows through an application, resulting in a more expressive solution that not only catches errors statically, but also achieves competitive run-time performance. In our preliminary evaluation, CircuitFlow outperforms the industry-leading Luigi library of Spotify by scaling better with the number of inputs. The innovative creation of CircuitFlow is also of note, exemplifying how to create a modular eDSL whose semantics necessitates effects, and where storing complex type information for program correctness is paramount.

Building Domain-Specific Machine Learning Workflows: A Conceptual Framework for the State-of-the-Practice

“…For chemistry students, Lafluente et al present an introductory workshop focusing on utilising Python and visualisation/ML libraries [66]. The example Jupyter notebooks 33 lead students through an introduction to 33 https://github.com/ML4chemArg/ Intro-to-Machine-Learning-in-Chemistry Python, basic statistics, exploratory data analysis, classification, and regression.…”

“…Another research avenue would be integration of model management approaches such as modelling variability and uncertainty techniques into workflow management tools [36]. The last research avenue we propose would be the integration of verification and va-lidity techniques such as recording performance metrics [22], enhancing type safety [33], and checking for formal properties such as reachability [35].…”