Impact of Selection Signature on Genomic Prediction and Heritability Estimation in Livestock

Natural or artificial selection could shape genetic architecture, e.g., the relationship between minor allele frequency (MAF) and the effect sizes of causal variants (CVs). This study aimed to investigate the impact of the MAF-effect size relationship (as a selection signature, <i>S</i>) on genomic prediction and heritability estimation in livestock, using both simulated data (Holstein) and real datasets (Holstein and pigs). We evaluated the performance of two models: (1) selection-adjusted genomic best linear unbiased prediction (GBLUP-S), and (2) MAF-stratified selection-adjusted genomic best linear unbiased prediction (GBLUP-SMS). Simulation results demonstrated that for traits under strong negative selection (<i>S</i> < -1), both GBLUP-S and GBLUP-SMS outperformed classic GBLUP. The prediction accuracy of GBLUP-S improved by 0.011-0.031, while GBLUP-SMS achieved a gain of 0.005-0.025. Furthermore, GBLUP-SMS exhibited lower sensitivity to variations in <i>S-</i>values, whereas GBLUP-S heavily relied on accurate <i>S</i> specification. When the true <i>S</i> was matched, GBLUP-SMS generated more unbiased (or comparable) heritability estimates and higher prediction accuracy relative to GBLUP-S. Critically, mismatched <i>S</i> in GBLUP-S led to increased bias in heritability estimates and reduced prediction accuracy. Cross-validation with real phenotypic data from Holsteins and pigs demonstrated that implementing selection-adjust methods improved prediction accuracy by 0.015 for FP in Holsteins and 0.01 for T1 in pigs, while enhancing the unbiasedness of heritability estimates across all traits. Negative selection signatures were identified for cattle (<i>S</i> = -0.5) and pig T1, T2, and T3 (<i>S</i> = -1.5, -1, and -2, respectively). These findings advance the theoretical framework of GBLUP-based genomic prediction and heritability estimation.