Sexual selection drives the speciation of lineages with contrasting mating systems

Repeated shifts in ecological strategies often lead to consistent speciation patterns across phylogenies. One example is the transition from outcrossing to self-fertilization in plants, which generally results in the reproductive isolation of the incipient selfing lineages. However, the evolutionary mechanisms driving their speciation remain poorly understood. In this study, we investigate the hybridization rate and barriers to gene flow between the recently evolved selfing lineage Capsella rubella and its outcrossing ancestor C. grandiflora. Through a survey of sympatric populations in Greece, we found that despite coexisting in the same habitats, the two species rarely form viable hybrids. Our investigation into the mechanisms underlying this reproductive isolation revealed that differences in the intensity of sexual selection between the lineages promote significant prezygotic isolation, with the strength of this isolation depending on the direction of gene flow. Traits that enhance male competitiveness in outcrossers decrease their chance of being pollinated by selfers, lowering the hybridization rate, but simultaneously increase the likelihood of selfers being pollinated by outcrossers. Selfers nevertheless limit hybridization through rapid and efficient self-fertilization mechanisms. Finally, postzygotic barriers, such as hybrid incompatibilities likely driven by differences in parental conflict intensity,¹ also contribute to the isolation of the two lineages. Therefore, shifts in the intensity of sexual selection and increase in selfing efficiency appear to be key drivers of reproductive isolation following mating system changes, possibly explaining recurrent speciation patterns in plant evolution.