Runs That Succeed and Are Wrong: An Audit of Silent Failures in a Multi-Omics Platform’s Validation Runs

Bioinformatics pipelines can finish with a success status, write parseable summaries and report wrong results. We audited the 42 validation runs of a multi-omics Nextflow platform that the author built, reading the summary archived for each run; every summary records the status “success”. Of the runs, 14 carry a confirmed silent failure in one of five classes, each traced to code. Bulk RNA-seq of fly, Arabidopsis and yeast was quantified against a default human transcript index, and fly and bacterial variants were annotated with the human SnpEff database. The proteomics pipeline’s false discovery rate (FDR) filter aborted on an option that OpenMS had renamed, and a shell fallback passed the unfiltered list on, as we reproduced in the OpenMS build behind its container. Metabolomics features were matched across samples on exact rounded keys, so that the matrix paired almost none of the features that replicate injections of one reference plasma share. And a spatial run filed as a brain section processed bronchoalveolar-lavage reads from another study. In the validation’s later round, its model judge gave the maximum score on every metric to answers that repeated the failures’ signatures: the expected answers came from the same summaries, and the metrics measure agreement with them, not whether a summary is true. A mapping-rate floor, which only five of the failed runs report, catches three. Checks written after the failures were known flag all 13 failures of the four classes they target, as they must, since a failure counts as confirmed only when its summary shows the signature; no check looks for reads from another study. Only clean runs test the checks, and no provenance, identity or invariant check flags any of the nine clean runs, a result that allows a false-positive rate as high as 34 percent; the checks behind eight of the 13 detections never ran on a clean run. On the run outputs, the checks catch the same failures and, after a fix to the audit’s strain table that had flagged a control, flag no clean run. How well the checks detect failures they were not written for is untested.