Teeth morphology in Red Sea fishes and comparative evolution of tooth-associated genes in herbivorous and carnivorous

Abstract Fish dentition exhibits substantial variability that reflects ecological niches and dietary specialization within the Red Sea ecosystem, highlighting adaptive evolutionary responses to different feeding strategies. This study investigated tooth morphology using scanning electron microscopy and explored the evolutionary basis of dental diversity through comparative genomic analyses. Ultrastructural examination of six representative fish families revealed distinct dentition patterns. Herbivorous (Scaridae and Siganidae) displayed incisiform, villiform, and molariform teeth, whereas carnivorous (Sphyraenidae, Holocentridae, Serranidae, Lethrinidae, and Lutjanidae) exhibited sharp conical and canine-like dentition adapted for prey capture. In parallel, 130 tooth-associated genes were analyzed across 25 teleost genomes representing herbivorous and carnivorous lineages. Phylogenetic analyses recovered well-supported family-level clades and suggested independent origins of herbivory consistent with convergent evolution. Selection analyses detected widespread episodic positive selection, with 70 genes remaining significant after correction. Branch-level analyses further identified reciprocal selection patterns between herbivorous and carnivorous lineages. Site-level analyses detected adaptive codons overlapping predicted functional regions in lrp5 and pam. Comparative tests further revealed contrasting selection pressures between trophic groups, with genes related to odontogenesis and skeletal development intensified in herbivores, whereas genes associated with growth factor signaling and neural processes were intensified in carnivores. Together, these findings highlight the interplay between dietary ecology, tooth morphology, and molecular evolution in teleost fishes. Functional validation will be required to confirm these genomic predictions.