HybTrans: A Molecular Generation and Optimization Model Integrating Protein Pocket Sequence and Structural Information

Deep learning models for drug discovery offer broader chemical space exploration and generate more novel molecules compared to traditional methods. Deep learning models currently used in drug design can be divided into two main categories: those based on ligands and those based on protein pockets. However, there is a lack of models that integrate both protein pocket sequence and structural information for molecular generation. This study proposes HybTrans, a molecular generation and optimization model that integrates protein pocket sequence and structural information. The model includes a sequence encoding module, a structure encoding module, a small molecule Self-Referencing Embedded Strings (SELFIES) representation module, a fusion decoding module, and a reinforcement learning module. The evaluation results demonstrate that the molecules produced by HybTrans have an average diversity of 0.815 and a docking affinity score of -7.828 kcal/mol (Vina), surpassing the performance of similar models. Additionally, the generated molecules exhibit high drug-likeness (QED), synthesizability (SA), and Lipinski’s rule of five scores (Lipinski). Ablation studies demonstrate the importance of key modules such as the fusion decoding module and the SELFIES representation module. A case study on the p21-activated kinase (PDB ID: 5i0b) binding pocket shows that HybTrans is capable of capturing the interactions between protein pockets and drug molecules, generating small molecules with high target affinity and excellent molecular properties.