240 Approximation of the logarithm of the determinant of high-dimensional matrices applied to animal breeding and genetics computations.

Abstract Evaluating the (restricted) log-likelihood of statistical models is widely used when estimating variance components and doing hypothesis testing in animal breeding and genetics. However, due to memory usage and extremely long computational times, performing likelihood evaluations is infeasible for large datasets. A major computational bottleneck in evaluating the likelihood is calculating the logarithm of the determinant (log-determinant) of high-dimensional matrices. This study aimed to present a method to approximate the log-determinant of positive semi-definite matrices, making likelihood evaluations feasible for very large datasets. Our method uses repeated matrix-vector products, notions of complex calculus, and contour integrals. We tested the approximation on beef and dairy cattle, chicken, and pig datasets, including both single and multiple-trait models. The average relative difference between approximated and exact log-determinant values was around 10-3. The approximation was 2 to 500 times faster than the exact calculation. Differences in computing time between the exact and approximated log-determinant increased with dataset size. For the largest dataset, with included 9,408,140 dairy cattle, 34,506 genotyped individuals, and a coefficient matrix rank of 19,170,955 from the mixed model equations, the exact calculation took 19 days, and the approximation took one hour. As an application example, we calculated the restricted likelihood and estimated variance components with expectation-maximization (EM), average information (AI) REML, and two derivative-free (DF) methods using the approximation of the log-determinant for a single-trait model. The exact and approximated log-likelihood showed maximums at the same heritability value among the REML methods. Our proposed approach lifts the computing time and memory limitations for likelihood evaluations in large datasets. The method allows approximating the log-determinant of positive semi-definite matrices, opening the possibility of performing likelihood evaluations for virtually any animal breeding dataset.