MCMC-CE: A Novel and Efficient Algorithm for Estimating Small Right-Tail Probabilities of Quadratic Forms with Applications in Genomics

Quadratic forms of multivariate normal variables play a critical role in statistical applications, particularly in genomics and bioinformatics. However, accurately computing small right-tail probabilities (<i>p</i>-values) for large-scale quadratic forms is computationally challenging due to the intractability of their probability distributions, as well as significant numerical constraints and computational burdens. To address these problems, we propose Markov chain Monte Carlo cross-entropy (MCMC-CE), an innovative algorithm that integrates MCMC sampling with the CE method, coupled with leading eigenvalue extraction and Satterthwaite-type approximation techniques. Our approach efficiently estimates small <i>p</i>-values for quadratic forms with their ranks exceeding 10,000. Through extensive simulation studies and real-world applications in genomics, including genome-wide association studies and pathway enrichment analyses, our method demonstrates advantageous numerical accuracy and computational reliability compared with existing approaches such as Davies', Imhof's, Farebrother's, Liu-Tang-Zhang's, and saddlepoint approximation methods. MCMC-CE provides a robust and scalable solution for accurately computing small <i>p</i>-values for quadratic forms, facilitating more precise statistical inference in large-scale genomic studies.