Taking Research to the Nextflow Level

How can a small eResearch platform level up their institute s computational workloads? At WEHI s Research Computing Platform (RCP), our goal is to make research as easy as possible. In 2022, one important area that needed improvement was in the area of bioinformatics workflows. Researchers needed an easy way to orchestrate large scale analyses, handle software dependencies, efficiently utilize the Slurm cluster, share their workflows with non-technical users and monitor workflow errors. This talk will discuss how we achieved all this using the Nextflow workflow manager. Improving Nextflow support was a gradual and organic process. One early milestone was configuring Nextflow for our Slurm HPC cluster, while another was the introduction of Nextflow training. The next large step came with a workflow engine comparison, where Nextflow won RCP s endorsement for new workflows. With this in place, we started a program of helping researchers adapt their pipelines from other frameworks and from ad hoc scripts. We will discuss a number of these workflow uplifts, with notable examples from WEHI s Advanced Genomics Facility. The introduction of Seqera Platform in 2023 broadened the userbase of our Nextflow pipelines, which RCP formalized with an access process and automated using the web API. Improving Nextflow support has successfully streamlined and simplified analyses for much of the institute, with usage continually growing. Now with 2 years of experience supporting Nextflow, we will also discuss some ongoing issues we have encountered, scenarios where Nextflow is not the best fit, and alternatives we have in mind for these scenarios.