Model Rankings in Drug–Target Affinity Prediction Vary Across Evaluation Regimes: A Controlled Multi-Seed Study of Model-Comparison Reliability

Abstract Drug–target affinity (DTA) prediction directly supports compound screening and target prioritization, but different deployment settings pose different generalization demands. We therefore asked whether conclusions from model comparisons remain reliable across evaluation regimes. Using cleaned KIBA (118 036 records) and Davis (30 056 records), we constructed warm, cold-target (defined by full protein sequence), and cold-drug settings under a unified protocol. Three approximately capacity-matched cross-modal fusion models and two controlled architectural baselines were compared with identical splits, training budgets, and evaluation procedures, using three paired model seeds. The study comprised 126 prespecified training runs. Held-out test sets were opened only after training and model selection were complete, with record-level, unique-input, and integrity audits providing end-to-end traceability. Relative performance and nominal rankings changed markedly across regimes: the three-seed mean C-index rankings for KIBA cold-target versus cold-drug gave Kendall’s τ = -1.00; the most consistent cross-dataset model-family differences emerged under cold-drug; and multi-seed analysis showed that a single run did not fully characterize the complete method ordering. A post-hoc VEGFR2 case, locked by performance-independent criteria before target-specific predictions were inspected, extended this pattern to compound prioritization for an unseen target. Together, these results support scenario-matched, multi-seed, auditable DTA model comparison as a more reliable basis for model selection and interpretation.