A Likelihood Ratio Test for Hybridization Under the Multispecies Coalescent

Several methods have been developed to carry out a statistical test for hybridization at the species level, including the ABBA-BABA test and HyDe. Here, we propose a new method for detecting hybridization and quantifying the extent of hybridization. Our test computes the likelihood of a species tree that is possibly subject to hybridization using site pattern frequencies from genomic-scale datasets under the multispecies coalescent. To do this, we extend the calculation of the likelihood for site pattern frequency data for the 4-taxon symmetric and asymmetric species trees proposed in Chifman and Kubatko (2015) by incorporating an inheritance parameter, resulting in efficient computation of the likelihood under a scenario of hybridization. We use this likelihood computation to construct a likelihood ratio test that a given species is a hybrid of two parental species. Simulations demonstrate that our test is more powerful than existing tests of hybridization, including HyDe, and that it achieves the desired type I error rate. We apply the method to two empirical data sets, one for which hybridization is believed to have occurred and one for which previous methods have failed to detect hybridization.