Sustainable data analysis with Snakemake

Abstract

Data analysis often entails a multitude of heterogeneous steps, from the application of various command line tools to the usage of scripting languages like R or Python for the generation of plots and tables. It is widely recognized that data analyses should ideally be conducted in a reproducible way. Reproducibility enables technical validation and regeneration of results on the original or even new data. However, reproducibility alone is by no means sufficient to deliver an analysis that is of lasting impact (i.e., sustainable) for the field, or even just one research group. We postulate that it is equally important to ensure adaptability and transparency. The former describes the ability to modify the analysis to answer extended or slightly different research questions. The latter describes the ability to understand the analysis in order to judge whether it is not only technically, but methodologically valid. Here, we analyze the properties needed for a data analysis to become reproducible, adaptable, and transparent. We show how the popular workflow management system Snakemake can be used to guarantee this, and how it enables an ergonomic, combined, unified representation of all steps involved in data analysis, ranging from raw data processing, to quality control and fine-grained, interactive exploration and plotting of final results.

Tracking the international spread of SARS-CoV-2 lineages B.1.1.7 and B.1.351/501Y-V2

“…All of the code underlying this daily lineage tracking web-report can be found at GitHub and Zenodo 12 . grinch is a python-based tool, the analysis pipeline of which is built on a snakemake backbone 13 . Every 24 hours a scheduled cron 14 task runs on our local servers.…”

RADSex: A computational workflow to study sex determination using restriction site‐associated DNA sequencing data

“…Runtime and peak memory usage of RaDsex (version 1.1.2) was measured with the "benchmark" directive of snakemake (Mölder et al, 2021) on the Genotoul computational platform using four threads.…”

Recovery of deleted deep sequencing data sheds more light on the early Wuhan SARS-CoV-2 epidemic

“…The computer code and input data necessary to reproduce all analyses described in this paper are available on GitHub at https://github .com/jbloom/SARS-CoV-2_PRJNA612766. This GitHub repository includes a Snakemake (Mölder et al 2021) pipeline that fully automates all steps in the analysis except for downloading of sequences from GISAID, which must be done manually as described in the GitHub repository's README in order to comply with GISAID data sharing terms.…”