Unified heterogeneity-aware benchmark of drug synergy prediction: a cross-study analysis of traditional machine learning and graph deep learning models

Drug synergy prediction holds great promise in accelerating combination therapy development and improving treatment efficacy in cancer and other complex diseases. However, progress in this area is hindered by considerable heterogeneity across experimental datasets, including variability in the number of drug combinations, inconsistencies in synergy scoring methodologies, and differences in data quality. Here, we present the first comprehensive benchmarking framework specifically designed to accommodate inter-dataset heterogeneity. This framework integrates 13 independent datasets encompassing 454,794 retained drug combination–cell line entries, 4247 drugs, and 187 cell lines, all of which are cancer cell lines. Our comparative evaluation of seven computational models for drug synergy prediction reveals that model performance strongly depends on both the scale and quality of the datasets. The graph model JointSyn performed favorably on datasets with larger numbers (> 10,000) of retained drug combinations, while the traditional random forest model performed competitively on smaller-scale datasets. We also observe that the ZIP scoring metric yields the highest accuracy in large-scale data, whereas HSA is more effective in sparse-data scenarios. However, different synergy metrics show significant variability in performance across datasets, suggesting that different synergy metrics capture distinct aspects of drug interactions, and the choice of metric can substantially affect model evaluation and cross‑dataset consistency. Furthermore, we find that well-designed small datasets can match or even surpass the performance of larger benchmarks, suggesting that different metrics are applicable to different datasets/testing scenarios. Our benchmark provides a robust foundation for fair model evaluation and paves the way for the development of more generalizable and preclinically relevant drug synergy prediction methods. We establish the first heterogeneity-aware benchmark for drug synergy prediction by integrating 13 datasets to quantify critical heterogeneity drivers. Through a comparative evaluation of 7 representative models, spanning traditional machine learning and state-of-the-art (SOTA) deep learning architectures, across 4 synergy scores, we compare the performance of graph-based and non-graph-based approaches under different benchmark settings and provide a comparative reference for metric behavior across datasets, collectively providing benchmarking standards to support future preclinical evaluation of synergy prediction models.