A Hybrid Ligand- and Structure-Based Drug Design Invoked through Square Attention in Protein–Ligand Transformer

Abstract Computer-aided molecular design based on evaluation of protein–ligand (PL) binding affinity is important for accelerating drug discovery. A deep-learning model for natural language processing (NLP) involving an attention mechanism (i.e., Transformer) is expected to improve the PL-affinity evaluation along with clarifying amino acid residues and ligand atoms that are responsible for molecular interactions. However, the original attention (i.e., a scaled dot-product with row-wise Softmax normalization) is an adapted method for relationships between homogeneous inputs (e.g., two sentences in the translation tasks) and thus is not suitable to analyze heterogeneity found in input data such as proteins (102–103 residues) and ligands (101–102 atoms). In fact, due to the significant difference between the numbers of protein residues and ligand atoms, gaps in the scale of attention weights emerge after the normalization, which thus affects model performance. Herein, we report PLTransformer, which involves our novel attention mechanism with Softmax normalization exerting on both directions of protein residues and ligand atoms in a more equivalent manner, for correcting the imbalance by a tandem of attention blocks with swapped inputs. As a result, our PLTransformer shows competitive affinity predictive performance over a representative baseline while also providing the ability to identify binding pockets, including allosteric sites, and important ligand scaffolds and protein substructures without employing three-dimensional (3D) structures of PL complexes (i.e., a ligand-based drug design (LBDD) framework). Thus, the possible PL (residue–ligand atom) contacts provided by PLTransformer can assist us to select plausible PL binding modes obtained by docking simulations, promoting the identification of the appropriate binding mode such as to maximize the performance of PLTransformer (i.e., structure-based drug design (SBDD) framework).