scBOA: Bayesian optimization of single-cell analysis pipelines for robust annotation and discovery

Accurate cell-type annotation in single-cell RNA sequencing is sensitive to upstream parameters often selected heuristically. We introduce scBOA, a two-stage framework that uses Bayesian optimization to select workflow parameters against a prespecified objective and reruns the selected configuration for reference-assisted analysis. Its configurable objectives combine annotation purity, marker concordance and transcriptomic cohesion, enabling fine-grained or conservative solutions. Optional extensions support low-confidence-cell refinement and multi-sample integration. Across simulations and fluorescence-activated cell sorting datasets of human peripheral blood mononuclear cells, pancreas and adult heart, we compare Gaussian-process scBOA with 14 alternative parameter-selection and clustering strategies. We show that performance depends on the objective, workflow, reference hierarchy and vocabulary, consensus-label construction and partition granularity. We find that scBOA is not uniformly superior, but shows strong aggregate performance when annotation targets align with biological structure. We apply a mouse Huntington’s disease dataset as an exploratory stress test of integration and reference mapping. Overall, we demonstrate that scBOA provides reproducible, objective-driven workflow optimization for reference-assisted single-cell analysis. Here, the authors introduce scBOA, a Bayesian optimization tool that helps researchers choose single-cell analysis settings for cell-type annotation. They perform benchmarking to show that performance depends on the dataset, reference and objective.