DPDispatcher: Scalable HPC Task Scheduling for AI-Driven Science

Artificial intelligence (AI) is reshaping computational science, but AI-driven workflows routinely span heterogeneous tasks executed across diverse high-performance computing (HPC) systems. We introduce DPDispatcher, an open-source Python framework for scalable, fault-tolerant task scheduling in such environments with an emphasis on lightweight submission, automatic retries, and robust resumption. DPDispatcher separates connection and file-staging concerns from scheduler control, supports multiple HPC job managers, and provides both local and secure shell (SSH) backends. DPDispatcher has been adopted by more than ten scientific packages. Representative use cases include active learning for machine-learning potentials, free-energy and thermodynamic integration workflows, large-scale materials screening, and large language model (LLM)-driven agents that launch HPC computations. Across these settings, DPDispatcher reduces operational overhead and error rates while improving portability and automation for reliable, high-throughput scientific computing.

Authors

Institutions

Publication Details

Journal
Journal of Chemical Information and Modeling
Published
2025-11-03
DOI
https://doi.org/10.1021/acs.jcim.5c02081
Primary Topic
Scientific Computing and Data Management
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

DPDispatcher: Scalable HPC Task Scheduling for AI-Driven Science

Cao Thang Nguyen, Tongqi Wen, Ping Tuo, Yaotang Zhang et al.
Journal of Chemical Information and Modeling
Scientific Computing and Data Management
article

DPDispatcher: Scalable HPC Task Scheduling for AI-Driven Science

Cao Thang Nguyen, Tongqi Wen, Ping Tuo, Yaotang Zhang, Yixiao Chen, Yun‐Pei Liu, Fengbo Yuan, Jinzhe Zeng, Mingkang Liu, Kai Cao, Zhaonan Ding, Yong‐Bin Zhuang, Haidi Wang, Jiahao Fan, Yibo Wang, Yifan Li, Handong Wang, Yuzhi Zhang, Linfeng Zhang, Jiameng Huang, Hao Yu
article en

Abstract

Artificial intelligence (AI) is reshaping computational science, but AI-driven workflows routinely span heterogeneous tasks executed across diverse high-performance computing (HPC) systems. We introduce DPDispatcher, an open-source Python framework for scalable, fault-tolerant task scheduling in such environments with an emphasis on lightweight submission, automatic retries, and robust resumption. DPDispatcher separates connection and file-staging concerns from scheduler control, supports multiple HPC job managers, and provides both local and secure shell (SSH) backends. DPDispatcher has been adopted by more than ten scientific packages. Representative use cases include active learning for machine-learning potentials, free-energy and thermodynamic integration workflows, large-scale materials screening, and large language model (LLM)-driven agents that launch HPC computations. Across these settings, DPDispatcher reduces operational overhead and error rates while improving portability and automation for reliable, high-throughput scientific computing.

Journal of Chemical Information and ModelingVol. 65(22)
Kanazawa University (JP), Boston University (US), University of Science and Technology of China (CN), Institute of Applied Physics (UA), Hefei University of Technology (CN), National University of Singapore (SG), Xiamen University (CN), Princeton University (US), Peking University (CN), Art Institute of Portland (US), Suzhou University of Science and Technology (CN), University of Alabama at Birmingham (US), Blue Planet (Norway) (NO), Institute of History of Art and Musicology (AT), Institute for Structural Research (PL), Computational Physics (United States) (US), SP Technology (South Korea) (KR), Institute of Applied Physics and Computational Mathematics (CN), Institution of Structural Engineers (GB), Collaborative Innovation Center of Chemistry for Energy Materials (CN), Preferred Networks (Japan) (JP), Ulsan National Institute of Science and Technology (KR), University of Hong Kong (HK), University of California, Berkeley (US)
Openalex Percentile: Top 70%
Scientific Computing and Data Management
DOISourceOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.