SWARM: Reimagining scientific workflow management systems in a distributed world
Modern scientific workflows process massive amounts of data from diverse instruments and sensors, leveraging geographically distributed, heterogeneous compute and storage resources—from leadership-class systems to edge devices—connected by high-performance networks. The diversity of resources introduces challenges in harnessing their full potential, with resilience issues arising across applications, system software, networks, storage, and hardware. Today, workflow management systems (WMS) coordinate the execution of computation and data management tasks across target resources. However, WMS’s centralized nature makes them vulnerable to faults and scalability issues that may result in failures of entire computational campaigns. This paper introduces a novel agentic framework for workflow management, fully distributing and decentralizing the WMS functions and modeling them as swarm intelligence agents infused with advanced artificial intelligence solutions and traditional distributed computing algorithms that can make coordinated decisions in the presence of failures of the underlying cyberinfrastructure.
Authors
- Komal Thareja (ORCID: https://orcid.org/0000-0002-8113-6967)
- Ewa Deelman (ORCID: https://orcid.org/0000-0001-5106-503X)
- Imtiaz Mahmud (ORCID: https://orcid.org/0000-0002-3390-2698)
- Kesheng Wu (ORCID: https://orcid.org/0000-0002-6907-3393)
- Franck Cappello (ORCID: https://orcid.org/0000-0002-7890-3934)
- Hongwei Jin (ORCID: https://orcid.org/0000-0002-2851-595X)
- Zizhong Chen (ORCID: https://orcid.org/0000-0003-2578-4940)
- Anirban Mandal (ORCID: https://orcid.org/0000-0001-5145-8618)
- Sheng Di (ORCID: https://orcid.org/0000-0002-7339-5256)
- Prasanna Balaprakash (ORCID: https://orcid.org/0000-0002-0292-5715)
- Shixun Wu (ORCID: https://orcid.org/0000-0001-8637-3307)
- Mariam Kiran (ORCID: https://orcid.org/0000-0001-5775-0855)
- Krishnan Raghavan (ORCID: https://orcid.org/0000-0001-9409-2011)
- Paweł Żuk (ORCID: https://orcid.org/0000-0001-5293-8517)
Institutions
- University of California, Riverside (US)
- University of Southern California (US)
- Argonne National Laboratory (US)
- Oak Ridge National Laboratory (US)
- Lawrence Berkeley National Laboratory (US)
- Renaissance Computing Institute (US)
Publication Details
- Journal
- The International Journal of High Performance Computing Applications
- Published
- 2025-05-15
- DOI
- https://doi.org/10.1177/10943420251339317
- Citations
- 3
- Primary Topic
- Scientific Computing and Data Management
- Type
- article
- Field-Weighted Citation Impact
- 8.34
Funders
- Advanced Scientific Computing Research