Effects of inbreeding on balancing selection: insights from Fisher's geometric model

Balancing selection is a potentially important factor in the maintenance of genetic variation for fitness and, alongside recessive deleterious mutations, the genetic basis of inbreeding depression. Classic population genetics theory predicts that inbreeding restricts the range of conditions leading to balancing selection. For example, in models of heterozygote advantage, the classic theory shows that inbreeding reduces the parameter conditions for balancing selection by a factor of 1-F, where F is Wright's inbreeding coefficient. However, without a model for the distribution of fitness effects of mutations or genotypes, this classic theory tells us little about the actual probability that new or segregating mutations meet criteria for balancing selection. Here, we develop an extension of Fisher's geometric model with which we explore how inbreeding affects the probability of balancing selection due to heterozygote advantage and its contribution to genetic variance for fitness. When taking the distribution of fitness effects among new, adaptive, and established mutations into account, we find that the prevalence of balancing selection is consistently, and often substantially, below the 1-F baseline implied by classic theory provided that most mutations have phenotypic effects that are small. The reduction is consistently greater for established mutations relative to adaptive mutations, which reinforces the idea that balanced genetic polymorphisms are far more likely to occur in outbred than inbred species. We discuss the implications of our results for studies of genetic variation for fitness and genome scans for signals of balancing selection.