A transfer-learned hierarchical variational autoencoder model for computational design of anticancer peptides.

e13665 Background: Cancer is a leading cause of mortality globally, necessitating the development of effective therapies. Anticancer peptides (ACPs) show promises given their selective targeting of cancer cells, low toxicity, and ability to overcome drug resistance. However, traditional discovery and optimization methods are time-consuming and expensive, underscoring the need for efficient computational strategies. Methods: We developed a Deep Hierarchical Conditional Variational Autoencoder (CVAE) for de novo ACP design, using transfer learning by initializing the ESM-2 pre-trained encoder. A comprehensive ACP dataset was collected from 9 different databases. Dataset contained 3,209 confirmed ACP and 4,292 non-ACP sequences. The non-ACP dataset was refined with the CD-HIT program, which clusters and removes redundant sequences, to keep those with < 40% sequence similarity to ACPs. Next, the CVAE encoder used local and global feature extractors to map peptides into a 256-dimensional latent space. The CVAE decoder reconstructed sequences using Gated recurrent unit (GRU) and Transformer layers with nucleus sampling. The CVAE multi-task classifier predicts anticancer and non-anticancer properties, guiding generation of highly active sequences. A gradual fine-tuning strategy was employed, progressively unfreezing the last 6 layers of ESM-2 encoder and applying discriminative learning rates with the AdamW optimizer. The model was trained using a combination of reconstruction loss, Kullback–Leibler (KL) divergence, and classification loss, optimized for balanced performance. For peptide generation, latent vectors were sampled using a Variational Gaussian Mixture Model, decoded into sequences, and filtered based on length, amino acid validity and anticancer probability (> 0.5). Results: The model shows robust performance in different domains. In classification, it achieved 0.89 accuracy, 0.88 precision, 0.87 recall, F1 of 0.87, and an area under receiver operating characteristic curve of 0.94 on the test set, indicating strong discriminative ability between ACPs and non-ACPs. Regarding generative capacity, the model successfully produced 100 unique anticancer peptides from 121 generation attempts, highlighting its ability to create viable candidates. In training, the loss functions consistently converged, the Fine-tune Loss and Reconstruction Loss decreased steadily, and the KL Loss remained stable to maintain meaningful latent representations affirming the model’s enhanced capability to reconstruct peptide sequences accurately. Conclusions: This study advances ACP design with a CVAE model integrating transfer learning and multi-task classification. The model's successful generation of new ACPs emphasize its potential to expedite the clinical translation and development of effective therapies.