Self-incompatibility and reticulate evolution contribute to interspecific compatibility of <i>Dendrobium</i>, a possible mechanism for promoting species diversity

Hybridization plays a crucial role not only in the origin and evolution of species but also in the formation of species diversity. <italic>Dendrobium</italic> is one of the largest genera in the Orchidaceae, exhibiting both a high rate of self-incompatibility (SI) and interspecific compatibility. However, there have been few reports on the distribution pattern of the SI and interspecific compatibility in <italic>Dendrobium</italic> and their relationship to the reticular evolution of <italic>Dendrobium</italic>. In this study, self- and cross-pollination experiments were conducted on selected <italic>Dendrobium</italic> species, and all previously reported data were integrated. The phylogenetic relationships of <italic>Dendrobium</italic> species were analyzed by using nuclear ITS sequence information and all published complete chloroplast genome data. SI and reticulate evolution were also studied for verification of their relationship to interspecific compatibility. The results showed that the SI rate in <italic>Dendrobium</italic> ranged from 55.64% to 70.96%, and species with self-incompatibility were widely distributed within the genus. <italic>Dendrobium</italic> species followed the SI × self-compatibility (SC) rule, suggesting SI affected the interspecific compatibility in <italic>Dendrobium</italic>. Further, when SC and SI/SC species served as parents, the rate of interspecific compatibility was higher than that with SI species as parents. The rate of interspecific compatibility (52.50%) between species or groups with closer reticulate evolutionary relationships was higher than that of all hybridization combinations (24.05%). Overall, this study provides new insights into the formation of high species diversity in <italic>Dendrobium</italic> and offers theoretical guidance for further hybridization breeding.