Distributed Data Analysis Workflow System across Multiple Data Hubs

In multisite data hub environments, data transfer costs and resource constraints challenge efficient data analysis workflow execution. Traditional analysis methods across multi-hubs often reduce performance and increase costs. To address these challenges, we introduce CROSS (Collaborative Resource-Oriented Scheduling System), which optimizes distributed workflows by utilizing data and resources across multiple hubs. CROSS enables collaboration between hubs, minimizes data transfer, and maximizes resource use through an efficient scheduling algorithm that considers data locality, resources, and workflow structure. Experiments with four scientific workflows show CROSS reduces makespan by up to 33.5% and improves CPU and memory efficiency by 1.58x and 1.59x, respectively, making it effective for multisite workflows.