Kill your shell scripts: Reproducible workflow management with Snakemake

You have a process to automate. It only has a couple of steps, one after the other, so you write a shell script. Then, you add more steps. Perhaps you want to cache some of the intermediary state, and only re-run when the underlying data change, to save time. And maybe what final stage you run depends on what output you’re looking for. Before long, you’re either writing a rat’s nest of ad-hoc code looking after these edge cases, or writing your own domain-specific language within Bash to handle this task of… Workflow Management. It turns out that you are far from the first person to have encountered the problem of workflow management, and there are many mature tools available to reduce your maintenance burden and let you focus on your specific workflows of interest. In this walkthrough I’ll introduce how to use Snakemake, a workflow manager developed in life science and commonly used in particle physics, but applicable for many purposes more widely. Topics I’ll cover include how to define rules that depend on the output of other rules, templated rules that can work for a variety of input files, and rules depending on some external metadata or configuration; how to parallelise running a workflow on a single machine; and how to run steps of a workflow using external computing resources such as an HPC cluster. Files: rsecon26-snakemake-gh-pages.zip: Contains the slides and template; the working copy of this is on GitHub slides.pdf: Contains a rendered version of the slides and speaker notes, in case GitHub Pages is down and you don't want to install NPM packages links.txt: The following links: The slides as rendered on GitHub Pages The Asciinema link at which the shell session will be streamed