Hierarchical Heuristic Species Delimitation Under the Multispecies Coalescent Model with Migration

The multispecies coalescent (MSC) model accommodates genealogical fluctuations across the genome and provides a natural framework for comparative analysis of genomic sequence data from closely related species to infer the history of species divergence and gene flow. Given a set of populations, hypotheses of species delimitation (and species phylogeny) may be formulated as instances of MSC models (e.g., MSC for 1 species versus MSC for 2 species) and compared using Bayesian model selection. This approach, implemented in the program bpp, has been found to be prone to over-splitting. Alternatively, heuristic criteria based on population parameters (such as population split times, population sizes, and migration rates) estimated from genomic data may be used to delimit species. Here, we develop hierarchical merge and split algorithms for heuristic species delimitation based on the genealogical divergence index (gdi) and implement them in a Python pipeline called hhsd. We characterize the behavior of the gdi under a few simple scenarios of gene flow. We apply the new approaches to a dataset simulated under a model of isolation by distance as well as 3 empirical datasets. Our tests suggest that the new approaches produced sensible results and were less prone to oversplitting. We discuss possible strategies for accommodating paraphyletic species in the hierarchical algorithm, as well as the challenges of species delimitation based on heuristic criteria.