Coupling geometric morphometrics and machine learning for mandibular sex estimation in Late Pleistocene and Late Modern populations

Accurate sex estimation is crucial for studying both modern and ancient human populations, yet methods are often limited to well-preserved skeletons. Here, we combine Geometric Morphometrics (GM) and Machine Learning (ML) to assess mandibular sexual dimorphism and classify sex across a wide chronological and geographic range to bracket the potential of this approach. Sixty-seven individuals from the modern, identified Luis Lopes collection (Portugal) and 18 Late Pleistocene individuals from Jebel Sahaba (Sudan) were surface scanned. Anatomical landmark coordinates were extracted and analyzed with GM, and ML models were trained on a subset of the modern sample to predict sex in both the remaining modern individuals and the Late Pleistocene specimens. GM revealed significant sexual dimorphism in all samples, and ML achieved high intrapopulation classification accuracy. However, predictions were less reliable when applied across the temporally and geographically distant Jebel Sahaba population, reflecting interpopulation differences in mandibular size and shape. These results demonstrate that while GM-ML approaches are powerful tools for sex estimation within populations, caution is required when extending models to other populations.