A framework for hyperparameter optimization of genomic cloud pipelines: improving the performance of GATK VETS variant filtration

Running complex machine-learning pipelines in the cloud is standard practice in genomics. However, optimizing these pipelines with respect to their hyperparameters can be challenging, which may limit their effectiveness. To address this gap, we developed an Optuna-based hyperparameter optimization framework and integrated it within the Terra cloud computing platform. To demonstrate our framework, we optimized the Genome Analysis Toolkit Variant Extract-Train-Score (GATK VETS) pipeline for genomic variant filtration. We evaluated the performance of our framework using both human and malaria genomic callsets, demonstrating improved accuracy in detecting sequencing artifacts. Our work underscores the importance of a flexible framework for tuning genomic cloud pipelines and highlights its potential for broader applicability in other genomic machine-learning analyses.