That's Not a Hybrid: How to Distinguish Patterns of Admixture and Isolation By Distance

Describing naturally occurring genetic variation is a fundamental goal of molecular phylogeography and population genetics. Popular methods for this task include STRUCTURE, a model-based algorithm that assigns individuals to genetic clusters, and principal component analysis (PCA), a parameter-free method. The ability of STRUCTURE to infer mixed ancestry makes it popular for documenting natural hybridisation, which is of considerable interest to evolutionary biologists, given that such systems provide a window into the speciation process. Yet, STRUCTURE can produce misleading results when its underlying assumptions are violated, like when genetic variation is distributed continuously across geographic space. To test the ability of STRUCTURE and PCA to accurately distinguish admixture from continuous variation, we use forward-time simulations to generate population genetic data under three demographic scenarios: two involving admixture and one with isolation by distance (IBD). STRUCTURE and PCA alone cannot distinguish admixture from IBD, but complementing these analyses with triangle plots, which visualise hybrid index against interclass heterozygosity, provides more accurate inference of demographic history, especially in cases of recent admixture. We demonstrate that triangle plots are robust to missing data, while STRUCTURE and PCA are not, and show that setting a low allele frequency difference threshold for ancestry-informative marker (AIM) identification can accurately characterise the relationship between hybrid index and interclass heterozygosity across demographic histories of admixture and range expansion. While STRUCTURE and PCA provide useful summaries of genetic variation, results should be paired with triangle plots before admixture is inferred.