TVAE: Transformer based Variational autoencoder for the design of drug molecules

Deep learning has transformed drug design by enabling the generation of novel molecular structures, with Variational Autoencoders (VAEs) playing a key role in learning latent representations. However, traditional VAEs often struggle to capture complex molecular dependencies. To address this, we propose a Transformer-based Variational Autoencoder (TVAE), integrating Transformer attention into the encoder to model intricate molecular patterns effectively. This approach leverages the self-attention mechanism to enhance the generation of diverse and valid molecules, crucial for drug discovery. TVAE achieves a novelty score of 0.941 ± 0.004 and the highest scaffold diversity (Scaf) score of 0.192 ± 0.008, outperforming baseline models. These results highlight TVAE’s ability to generate structurally diverse and novel molecules, demonstrating the potential of combining Transformers with VAEs for advancing generative models in drug design and material discovery..