Human population dynamics in Upper Paleolithic Europe inferred from fossil dental phenotypes

Despite extensive archaeological research, our knowledge of the human population history of Upper Paleolithic Europe remains limited, primarily due to the scarce availability and poor molecular preservation of fossil remains. As teeth dominate the fossil record and preserve genetic signatures in their morphology, we compiled a large dataset of 450 dentitions dating between ~47 and 7 thousand years ago (ka), outnumbering existing skeletal and paleogenetic datasets. We tested a range of competing demographic scenarios using a coalescent-based machine learning Approximate Bayesian Computation (ABC) framework that we modified for use with phenotypic data. Mostly in agreement with but also challenging some of the hitherto available evidence, we identified a population turnover in western Europe at ~28 ka, isolates in western and eastern refugia between ~28 and 14.7 ka, and bottlenecks during the Last Glacial Maximum. Methodologically, this study marks the pioneering application of ABC to skeletal phenotypes, paving the way for exciting future research avenues.