Enhancing protein–ligand pose prediction via dual-branch deep learning with docking-oriented multi-component loss and iterative refinement

Abstract Protein-ligand pose prediction is a core task in structure-based drug discovery because it determines how a ligand fits within a protein pocket and directly affects downstream virtual screening and lead-optimization workflows. Recent graph neural network (GNN) methods have shown promise for protein-ligand pose prediction, while improving the accuracy of an initial docked pose remains an important task. In this work, we present KTransPose, a GNN-based protein-ligand pose-refinement framework that combines a dual-branch architecture, a docking-oriented multi-component loss, and iterative refinement. The dual-branch architecture combines a global pose transformation branch that predicts rotation and translation with a local coordinate correction branch that predicts atom-wise coordinate adjustments. The multi-component loss combines root mean squared distance (RMSD) when computed without rotational or translational alignment, mean squared error (MSE) after alignment by the Kabsch algorithm, intraligand distance regularization, and a protein-ligand clash penalty. We instantiate the framework with several GNN architectures and find that a TransformerConv-based model provides the strongest overall performance. Using the same preprocessing, initial docked poses, and evaluation procedure, KTransPose improves upon MedusaGraph on both PDBbind-2020 and CASF-2016 benchmark datasets, reducing the mean RMSD from 5.08 Å to 4.47 Å and from 4.81 Å to 4.24 Å respectively, corresponding to improvements of approximately 12% in both datasets.