Predicting maize hybrid performance with machine learning and a locus-specific weighted degree of dominance transformation

The genetic architecture of a trait plays a vital role in the predictive ability of genomic models. While classical methods such as genomic best linear unbiased prediction (GBLUP) remain widely used in plant breeding, the value of machine learning (ML) is increasing because of its ability to capture non-linear effects. This study assessed ML and classical models incorporating a locus-specific weighted dominance effect transformation matrix for genomic prediction in hybrid maize. We evaluated five models in simulated and real hybrid maize dataset: (1) XGBoost with transformed SNP marker matrix (ML_Transformed), (2) XGBoost with conventional SNP marker matrix (ML), (3) GBLUP with additive effects only (AM), (4) GBLUP with additive and dominance effects (ADM), and (5) GBLUP with transformed SNP marker matrix (CADM). Two hybrid maize dataset were simulated, polygenic and oligogenic with dominance levels ranging from 0% to 40% while the real maize hybrid dataset evaluation consisted of traits with diverse genetic architectures, including grain yield, test weight, ear height, plant height, pollen and silk days after planting, and grain moisture. Results showed that the dominance transformation had mixed effects: it did not enhance ML performance, but only improved CADM in simulated scenarios. Across both simulated and real data, ML generally exceeded GBLUP performance, except in polygenic simulations where CADM outperformed all other models including the ML models. We also found that increasing dominance levels generally reduced predictive accuracy, regardless of the model. In general, these results suggest that CADM and ML_Transformed are promising for application in plant breeding. However, their success depends on the underlying traits genetic architecture, highlighting the importance of dominance-incorporating and trait-adaptable approaches to genomic prediction for optimizing breeding strategies.