Directed Acyclic Graph on Cross-Application Programmable I/O: Adding streaming flavour to scientific workflows

This paper introduces DAGonCAPIO, a workflow framework that integrates the DAGonStar engine with the CAPIO middleware to enable I/O streaming in scientific workflows. DAGonStar utilizes a Directed Acyclic Graph (DAG) to orchestrate tasks. At the same time, CAPIO enables downstream tasks to process data as soon as partial outputs become available, without requiring modifications to application code. This integration reduces delays from traditional file-based communication. The system uses the workflow:// schema to define data dependencies and generate CAPIO coordination scripts. DAGonStar was modified to support early scratch directory naming and decoupled task execution. Experiments with a WRF-based weather forecasting workflow on a 256-core cluster demonstrate that DAGonCAPIO reduces time-to-first-result by up to 4171 seconds, achieving a nearly 10x speedup.