Covariance reaction norms: A flexible method for estimating complex environmental effects on trait (co)variances

Abstract Estimating quantitative genetic and phenotypic (co)variances is crucial for investigating evolutionary ecological phenomena such as developmental integration, life history trade‐offs and niche specialization, as well as for describing selection and predicting multivariate evolution in the wild. While most studies assume (co)variances are fixed over short timescales, environmental heterogeneity can rapidly modify the variation of and associations among organisms' traits. Here I extend prior multilevel regression models for quantitative genetic inference (so‐called animal models) to develop a novel covariance reaction norm (CRN) model, which can be used to detect how trait (co)variances respond to continuous, multivariate, and potentially nonlinear environmental change, even in the absence of repeated individual measurements or experimental breeding designs. After introducing the CRN model, I use simulations to validate its implementation for Bayesian inference in Stan, as well as to compare its performance to standard character state and random regression approaches. Findings demonstrate superior accuracy and power for detecting environmental effects on genetic covariance with modest sample sizes. I then apply the CRN model to long‐term field data on cooperation among meerkats ( Suricata suricatta ). I find nonlinear effects of group size on the genetic (co)variances of cooperative behaviours, leading to increased social niche specialization among foraging and pup feeding versus babysitting tasks in larger groups. Multivariate gene‐by‐environment interactions are also observed in response to age, sex and dominance status. R code and a tutorial are provided to aid empiricists in applying CRN models to their own datasets.