Independent Benchmarking of Short-Read Metagenomics Pipelines

The generation of metagenomics data has increased exponentially within the last ten years, supported by the rapid evolution of next generation sequencing techniques for both short and long reads. Accompanying this thriving period of available information, many challenges have raised such as the establishment of data management plans, developing more efficient software to analyze data and boosting reproducibility by setting proper bioinformatics working conditions. Our main goal was to perform an independent benchmark study among cutting edge short-reads metagenomics pipelines to establish real case scenario comparisons, as well as provide important insights to consider for selecting the tool that suits study-related specific requirements. To achieve this objective, we studied several public metagenomics datasets obtained from a wide variety of crop soils and rhizosphere including rice, wheat and maize through a wide variety of pipelines for Metagenome Assembled Genomes (MAGs) reconstruction, taxonomical annotation and functional analysis. The chosen pipelines were nf-core/mag, metagenome-ATLAS, MAGNETO, DATMA and MetaWRAP. Several differences were found among the pipelines in terms of number of reconstructed MAGs, completeness, contamination, functional characterization and taxonomical distribution. Among the most important results, enhanced taxonomical resolution was achieved by pipelines allowing for co-assembly, i.e., nf-core/mag and MAGNETO. In regards to performance while run in an HPC cluster, pipelines relying on workflow managers, e.g. Snakemake or Nextflow, depicted a better behavior by obtaining always the same number of MAGs along with their features, improving reproducibility given their easiness in parallelizing tasks; however, in spite of the simplicity reached by the organized workflow within these pipelines, many issues related to installation, dependencies and downloading databases were found across pipelines. Considering these outcomes, this work constitutes an important independent comparison study among recently released metagenomics pipelines, highlighting their advantages and disadvantages, and empowering new users with useful criteria to select the pipeline best adapted to their requisites. Finally, an extended analysis including long reads, and pipelines able to handle them, should be considered to enrich the comparison among pipelines and their power to achieve more accurate results.