Phylogenetic patterns and genomic correlates of pronounced neocortical reduction in New World monkeys

Introduction Primate brain evolution is characterized by neocortical expansion, yet secondary reductions in brain size have occurred in several lineages. This study reconstructs the evolutionary dynamics of relative neocortex size in anthropoid primates and explores genomic correlates of identified neocortical reductions in New World monkeys to investigate whether such changes reflect direct selection on neural traits or arise from other mechanisms. Methods Neocortex and medulla volumes were compiled for anthropoid species. Phylogenetic modeling of relative neocortex size was performed using Fabric-regression with medulla volume as covariate. Genomic analyses involved codon-based tests for positive selection (aBSREL) and shifts in selection intensity (RELAX) across 18,177 orthologous protein-coding genes, followed by functional enrichment and characterization of candidate genes. Results Phylogenetic analyses revealed heterogeneous changes in relative neocortex size across anthropoids, with a statistically significant reduction along the ancestral Callitrichinae + Aotus branch. Genomic scans identified 12 genes under positive selection, two under intensified and none under relaxed selection on this branch, with no enrichment for neurodevelopmental or brain-related pathways. Only a single candidate with potential neurodevelopmental relevance was identified (ZNF512B). Conclusion The absence of a clear neurodevelopmental genomic signal contrasts with previously reported signatures of positive selection on brain-related genes in encephalized cebids, suggesting that brain expansions and reductions in anthropoids may not share a common genomic basis. This also opens up the possibility that the neocortical reduction detected here emerged as a by-product of selection on other traits (e.g. body size or life-history), rather than as a direct target of selection on neural phenotypes.