<scp>FlowPilot:</scp> A Suggestion System for Designing Scientific Workflows

Scientific Workflows (SWF) encapsulate data processing tasks by organizing various tools, operators, and data in a logical flow. Due to their complex and domain-specific nature, developing SWFs remains laborious. Current code-generating large language models (Code LLMs) struggle to assist users in developing these workflows. This limitation arises primarily from the insufficient availability of relevant training data in public repositories, making it challenging for LLMs to learn specialized patterns and domain-specific logic. To address this, we propose FlowPilot , a suggestion framework for developing SWFs that assists developers by suggesting the next operator. Our system complements Code LLMs by enriching the source code to generate more accurate results. FlowPilot leverages a similarity knowledge base (SKB) that indexes historical workflows to find the ones matching the current context. To generate relevant suggestions, FlowPilot employs a statistical approach based on Markov chains to identify the most likely next step. As a proof of concept, we evaluated our system on NextFlow workflows and the results demonstrate the effectiveness of FlowPilot . It outperforms state-of-the-art code-generating models, e.g., Llama-4, and traditional methods, e.g., association rule mining techniques.

Authors

Institutions

Publication Details

Journal
Proceedings of the ACM on Management of Data
Published
2026-04-02
DOI
https://doi.org/10.1145/3786653
Primary Topic
Scientific Computing and Data Management
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

FlowPilot: A Suggestion System for Designing Scientific Workflows

Matthias Weidlich, Mahdi Esmailoghli
Proceedings of the ACM on Management of Data
Scientific Computing and Data Management
article

FlowPilot: A Suggestion System for Designing Scientific Workflows

Matthias Weidlich, Mahdi Esmailoghli
article en

Abstract

Scientific Workflows (SWF) encapsulate data processing tasks by organizing various tools, operators, and data in a logical flow. Due to their complex and domain-specific nature, developing SWFs remains laborious. Current code-generating large language models (Code LLMs) struggle to assist users in developing these workflows. This limitation arises primarily from the insufficient availability of relevant training data in public repositories, making it challenging for LLMs to learn specialized patterns and domain-specific logic. To address this, we propose FlowPilot , a suggestion framework for developing SWFs that assists developers by suggesting the next operator. Our system complements Code LLMs by enriching the source code to generate more accurate results. FlowPilot leverages a similarity knowledge base (SKB) that indexes historical workflows to find the ones matching the current context. To generate relevant suggestions, FlowPilot employs a statistical approach based on Markov chains to identify the most likely next step. As a proof of concept, we evaluated our system on NextFlow workflows and the results demonstrate the effectiveness of FlowPilot . It outperforms state-of-the-art code-generating models, e.g., Llama-4, and traditional methods, e.g., association rule mining techniques.

Proceedings of the ACM on Management of DataVol. 4(1)
Humboldt-Universität zu Berlin (DE)
Deutsche Forschungsgemeinschaft
Industry, innovation and infrastructure
Openalex Percentile: Top 15%
Scientific Computing and Data Management
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.