An order-aware dataflow model for parallel Unix pipelines

Handa, Shivam; Καλλάς, Κωνσταντίνος; Vasilakis, Nikos; Rinard, Martin

doi:10.1145/3473570

Cited by 3 publications

(3 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Existing work has pushed towards improved performance through data parallelism ( [14,23,24]) and distribution ( [6,19]). Approaches in this area rely on annotations of commonly used commands, which the optimizers/planners use to rearrange programs.…”

Section: Discussionmentioning

confidence: 99%

Report on the "The Future of the Shell" Panel at HotOS 2021

Greenberg,

Kallas,

Vasilakis

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

This document summarizes the challenges and possible research directions around the shell and its ecosystem, collected during and after the HotOS21 Panel on the future of the shell. The goal is to create a snapshot of what a number of researchers from various disciplines-connected to the shell to varying degrees-think about its future. We hope that this document will serve as a reference for future research on the shell and its ecosystem. WHAT WAS "THE FUTURE OF THE SHELL" PANEL?Michael Greenberg, Konstantinos Kallas, and Nikos Vasilakis organized and ran a panel titled "The Future of the Shell: Unix and Beyond" at HotOS XVIII on June 3rd, 2021, from 1pm to 2:30pm ET [10]. The entire event was virtual due to COVID-19, held on Zoom (with support from Slack and a shared Google Doc for notes).The goal of the event was to suggest the shell as an area worthy of attention and research, using discussions to find themes and problems. As a conversation starter, the event was closely coupled to a position paper published by the panel's organizers in the same venue and presented the day before, highlighting key recent enablers and open problems in the shell [11]. By bringing together researchers and engineers from disparate communities-e.g., systems, languages, security-the event aimed to explore and highlight impactful research directions around the shell.

show abstract

Section: Discussionmentioning

confidence: 99%

Report on the "The Future of the Shell" Panel at HotOS 2021

Greenberg,

Kallas,

Vasilakis

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…This sequential execution often leaves available data parallelism, in which a command operates on different parts of an input stream in parallel, unexploited. This observation has motivated the development of systems that exploit data parallelism in shell pipelines [5,10,14]. A key prerequisite is obtaining the combiners necessary for merging the resulting multiple parallel output streams correctly into a single output.…”

Section: Introductionmentioning

confidence: 99%

“…A key prerequisite is obtaining the combiners necessary for merging the resulting multiple parallel output streams correctly into a single output. Previous systems rely on developers to manually implement such combiners and associate them with their corresponding shell commands [5,10,14].…”

Section: Introductionmentioning

confidence: 99%

Automatic synthesis of parallel unix commands and pipelines with KumQuat

Shen

Rinard

Vasilakis³

2022

Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming

View full text Add to dashboard Cite

We present KumQuat, a system for automatically generating data-parallel implementations of Unix shell commands and pipelines. The generated parallel versions split input streams, execute multiple instantiations of the original pipeline commands to process the splits in parallel, then combine the resulting parallel outputs to produce the final output stream. KumQuat automatically synthesizes the combine operators, with a domain-specific combiner language acting as a strong regularizer that promotes efficient inference of correct combiners. We present experimental results that show that these combiners enable the effective parallelization of our benchmark scripts. CCS Concepts: • Computing methodologies → Concurrent computing methodologies; • Software and its engineering → Software development techniques.

show abstract

An empirical investigation of command-line customization

Schröder

Cito

2021

Empir Software Eng

View full text Add to dashboard Cite

The interactive command line, also known as the shell, is a prominent mechanism used extensively by a wide range of software professionals (engineers, system administrators, data scientists, etc.). Shell customizations can therefore provide insight into the tasks they repeatedly perform, how well the standard environment supports those tasks, and ways in which the environment could be productively extended or modified. To characterize the patterns and complexities of command-line customization, we mined the collective knowledge of command-line users by analyzing more than 2.2 million shell alias definitions found on GitHub. Shell aliases allow command-line users to customize their environment by defining arbitrarily complex command substitutions. Using inductive coding methods, we found three types of aliases that each enable a number of customization practices: Shortcuts (for nicknaming commands, abbreviating subcommands, and bookmarking locations), Modifications (for substituting commands, overriding defaults, colorizing output, and elevating privilege), and Scripts (for transforming data and chaining subcommands). We conjecture that identifying common customization practices can point to particular usability issues within command-line programs, and that a deeper understanding of these practices can support researchers and tool developers in designing better user experiences. In addition to our analysis, we provide an extensive reproducibility package in the form of a curated dataset together with well-documented computational notebooks enabling further knowledge discovery and a basis for learning approaches to improve command-line workflows.

show abstract

An order-aware dataflow model for parallel Unix pipelines

Cited by 3 publications

References 59 publications

Report on the "The Future of the Shell" Panel at HotOS 2021

Report on the "The Future of the Shell" Panel at HotOS 2021

Automatic synthesis of parallel unix commands and pipelines with KumQuat

An empirical investigation of command-line customization

Contact Info

Product

Resources

About