Enhancing Diversity of Template-Free Retrosynthesis Prediction via Hierarchical Latent Variables

Retrosynthesis aims to identify sets of reactants capable of synthesizing a target molecule and has recently benefited from advancements in template-free sequence-translation models, which offer both efficiency and high predictive accuracy. A challenge in this domain is effectively capturing the intrinsic one-to-many relationship characteristic of chemical reactions. To address this, we propose a Hierarchical Conditional Variational Auto-Encoder (HCVAE) module that can be seamlessly integrated into existing template-free retrosynthesis frameworks. Our method establishes a hierarchical latent space that transitions from continuous to discrete representations: a continuous latent variable explores diverse chemical transformation proposals, while a discrete latent variable groups them into high-level reaction classes. This design links one product to multiple possible reactants, thereby enhancing coverage of multicandidate synthesis schemes. Extensive evaluations conducted on three publicly available benchmarks, encompassing both single-step prediction and multistep planning tasks, demonstrate that the HCVAE consistently improves performance across various backbone architectures. For instance, the single-step RootAligned model exhibits an increase in top-10 exact match accuracy on the USPTO-50k data set from 90.5% to 91.6%, meanwhile the DirectMultistep model shows improvements from 49.3% to 53.1% and from 43.0% to 46.7% on the <i>n</i>₁ and <i>n</i>₅ sets of the PaRoutes data set, respectively. Further analyses indicate that the learned latent space organization provides a structured mechanism for navigating alternative reaction proposals and facilitates practical multistep synthesis of drug-like molecules.