Severe bottleneck of ancient <i>Homo</i> populations: Insights from computational modeling and relevant fossil evidence

Abstract Reconstructing ancient population size history is essential for understanding the evolutionary origin of Homo sapiens. We recently developed the fast infinitesimal time coalescent process (FitCoal) and detected a severe population bottleneck occurring approximately 930 thousand years ago. However, two recent studies compared FitCoal and mushi and concluded that the severe bottleneck is a statistical artifact. In this study, we compared the two methods against a benchmark of ten billion msprime coalescent simulations. We demonstrate that FitCoal achieves both superior speed and accuracy in expected site frequency spectrum (SFS) estimation. Analyses of simulated datasets confirmed that FitCoal reliably recovers the bottleneck, whereas mushi fails under identical conditions. Independent fossil and paleoclimate evidence is consistent with the timing and evolutionary impact of this bottleneck, including associations with hominin dispersals, speciation events, and a subsequent increase in brain size. These findings refine the demographic history of Homo during the Pleistocene and highlight the importance of high-precision SFS computation for revealing critical evolutionary transitions that shaped modern human ancestry.