Amhy/Amhr2y-mediated sex determination in two distantly related teleosts relies on the conserved Alk3-Smad5 axis

In teleosts, homologs of Anti-Müllerian Hormone (Amhy) and its type II receptor (Amhr2/Amhr2y) have been independently recruited as master sex-determination genes in about 50% of known cases. However, it remains unknown whether a conserved transducer pair exists, as the requisite type I receptors and R-Smad effectors remain unidentified amidst their diversity and potential redundancy. In this study, we employed an in vitro reporter assay to screen five type I receptors (Alk2a, Alk2b, Alk3, Alk6a, Alk6b) and three R-Smads (Smad1, Smad5, Smad8), discovering that only Alk3, Alk6a, or Alk6b, in combination with Smad5, significantly activated Amhy/Amhr2 signaling. In Nile tilapia, levels of phosphorylated Smad5 (p-Smad5) were notably elevated in XY gonads compared with XX gonads during the critical sex-determination window (8 to 15 dpf), while total Alk3 and Smad5 expression did not exhibit sexual dimorphism. The inhibition of type I receptors in XY fish resulted in feminization or complete sex reversal. Similarly, CRISPR/Cas9 mutagenesis of alk3 or smad5 led to male-to-female sex reversal in F0 mosaic mutants. Importantly, homozygous mutations in alk3 or smad5 resulted in embryonic lethality at the gastrula stage, whereas mutations in other type I receptors or R-Smads were viable and demonstrated normal sexual development. The conservation of this pathway was further substantiated in Southern catfish, where mutations in alk3a or smad5 also induced sex reversal in XY individuals. Collectively, our findings establish Alk3 and Smad5 as essential and specific transducers of the Amhy/Amhr2-mediated sex-determination pathway, revealing a potentially conserved signaling axis across teleosts.