CATS: context-aware timeline scheduler for data pipelines on the computing continuum

Abstract The increasing adoption of the Internet of Things (IoT) is fuelling the expanded use of data pipelines (DPs) for data processing within Edge, Fog, and Cloud Computing Continuum infrastructures. Modern DPs, represented as directed acyclic graphs, consist of heterogeneous steps that process data to generate business value. In practice, DPs cannot be benchmarked in advance, which prevents the use of traditional scheduling approaches. To address this, we introduce the context-aware timeline scheduler (CATS), a novel approach that leverages pipeline context through dry runs with low-scale input volumes (as opposed to using benchmark data) to schedule data pipelines. CATS aims to optimize makespan and cost for large-scale deployments on Computing Continuum resources. Dry runs consist of consecutive testing deployments of pipeline steps on the target infrastructure, providing measurements of CPU, memory usage, and execution time. Based on these dry runs, data locality information and pipeline structure, CATS predicts the resource requirements, time of execution, and level of horizontal scaling for each step for larger total input volumes. Using the predictions, it computes a timeline of scheduling events to be executed when orchestrating the pipeline deployment on target infrastructure. We evaluated CATS using an industrial spot welding data preparation pipeline for downstream AI/ML-based predictive analytics, which was tested on real infrastructure that emulates a Fog/Cloud Continuum. The experimental results show accuracy for both predicting resource usage and makespan for target volumes that are up to two orders of magnitude higher than what was measured in the dry-run data.