A heterogeneity-aware parallel method for radio astronomy data transfer

Radio astronomy observations generate massive heterogeneous datasets with file sizes ranging from kilobytes to gigabytes and total volumes reaching terabytes per session. Telescope sites in remote areas face limited network connectivity, making efficient data transfer a significant challenge. The conventional rsync tool, while widely used for incremental transfer, suffers from serial limitations that prevent full bandwidth utilization, particularly for extreme file-size distributions characteristic of radio astronomy data. This paper presents a heterogeneity-aware parallel data transfer method built upon the rsync framework, designed for datasets from facilities such as the Qitai radio telescope (QTT). Our approach introduces three key contributions: (i) a heterogeneity-aware task partitioning strategy with an adaptive correction factor κ ( C v ) that isolates files into singleton chunks; (ii) a mathematical performance model quantifying the impact of data heterogeneity on parallel speedup through a load imbalance factor γ ( N , C v , ρ ) ; and (iii) a checkpoint-based fault tolerance mechanism that eliminates costly directory re-traversal overhead upon interruption. Experimental evaluation on both simulated and authentic ASKAP observation data demonstrates the effectiveness of our method. On the representative data-mixed workload, our method outperforms Optimized Rsync by 16.8% on simulated data and by 28.8% on real data, while maintaining CPU utilization below 1.14% and a memory footprint approximately three times smaller than Standard Rsync. The efficiency ratio shows at least two orders of magnitude higher resource efficiency than all baseline tools. The proposed method offers an alternative transmission solution for the QTT data transfer system, seamlessly integrating with the existing rsync ecosystem while consuming minimal computing resources.

Authors

Institutions

Publication Details

Journal
Astronomy and Computing
Published
2026-11-16
DOI
https://doi.org/10.1016/j.ascom.2026.101219
Primary Topic
Distributed and Parallel Computing Systems
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A heterogeneity-aware parallel method for radio astronomy data transfer

Hongmei Tang, Wen-na Cai, Hailong Zhang, Jie Wang et al.
1 links
Astronomy and Computing
Distributed and Parallel Computing Systems
article

A heterogeneity-aware parallel method for radio astronomy data transfer

Hongmei Tang, Wen-na Cai, Hailong Zhang, Jie Wang, Su Shi, Ya-Zhou Zhang, Xinchen Ye, Yuyue Jiao, Ting ZHANG, Wan-qiong Wang, Jia Li, Bo Wang
article en

Abstract

Radio astronomy observations generate massive heterogeneous datasets with file sizes ranging from kilobytes to gigabytes and total volumes reaching terabytes per session. Telescope sites in remote areas face limited network connectivity, making efficient data transfer a significant challenge. The conventional rsync tool, while widely used for incremental transfer, suffers from serial limitations that prevent full bandwidth utilization, particularly for extreme file-size distributions characteristic of radio astronomy data. This paper presents a heterogeneity-aware parallel data transfer method built upon the rsync framework, designed for datasets from facilities such as the Qitai radio telescope (QTT). Our approach introduces three key contributions: (i) a heterogeneity-aware task partitioning strategy with an adaptive correction factor κ ( C v ) that isolates files into singleton chunks; (ii) a mathematical performance model quantifying the impact of data heterogeneity on parallel speedup through a load imbalance factor γ ( N , C v , ρ ) ; and (iii) a checkpoint-based fault tolerance mechanism that eliminates costly directory re-traversal overhead upon interruption. Experimental evaluation on both simulated and authentic ASKAP observation data demonstrates the effectiveness of our method. On the representative data-mixed workload, our method outperforms Optimized Rsync by 16.8% on simulated data and by 28.8% on real data, while maintaining CPU utilization below 1.14% and a memory footprint approximately three times smaller than Standard Rsync. The efficiency ratio shows at least two orders of magnitude higher resource efficiency than all baseline tools. The proposed method offers an alternative transmission solution for the QTT data transfer system, seamlessly integrating with the existing rsync ecosystem while consuming minimal computing resources.

Astronomy and ComputingVol. 58
Chinese Academy of Sciences (CN), Xinjiang Astronomical Observatory (CN), University of Chinese Academy of Sciences (CN)
Industry, innovation and infrastructure
Distributed and Parallel Computing Systems
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon