A heterogeneity-aware parallel method for radio astronomy data transfer
Radio astronomy observations generate massive heterogeneous datasets with file sizes ranging from kilobytes to gigabytes and total volumes reaching terabytes per session. Telescope sites in remote areas face limited network connectivity, making efficient data transfer a significant challenge. The conventional rsync tool, while widely used for incremental transfer, suffers from serial limitations that prevent full bandwidth utilization, particularly for extreme file-size distributions characteristic of radio astronomy data. This paper presents a heterogeneity-aware parallel data transfer method built upon the rsync framework, designed for datasets from facilities such as the Qitai radio telescope (QTT). Our approach introduces three key contributions: (i) a heterogeneity-aware task partitioning strategy with an adaptive correction factor κ ( C v ) that isolates files into singleton chunks; (ii) a mathematical performance model quantifying the impact of data heterogeneity on parallel speedup through a load imbalance factor γ ( N , C v , ρ ) ; and (iii) a checkpoint-based fault tolerance mechanism that eliminates costly directory re-traversal overhead upon interruption. Experimental evaluation on both simulated and authentic ASKAP observation data demonstrates the effectiveness of our method. On the representative data-mixed workload, our method outperforms Optimized Rsync by 16.8% on simulated data and by 28.8% on real data, while maintaining CPU utilization below 1.14% and a memory footprint approximately three times smaller than Standard Rsync. The efficiency ratio shows at least two orders of magnitude higher resource efficiency than all baseline tools. The proposed method offers an alternative transmission solution for the QTT data transfer system, seamlessly integrating with the existing rsync ecosystem while consuming minimal computing resources.
Authors
- Hongmei Tang (ORCID: https://orcid.org/0000-0001-5918-3436)
- Wen-na Cai (ORCID: https://orcid.org/0009-0003-2671-1556)
- Hailong Zhang (ORCID: https://orcid.org/0000-0002-8951-7094)
- Jie Wang (ORCID: https://orcid.org/0000-0003-0380-6395)
- Su Shi (ORCID: https://orcid.org/0009-0006-7087-8638)
- Ya-Zhou Zhang (ORCID: https://orcid.org/0000-0001-6046-2950)
- Xinchen Ye (ORCID: https://orcid.org/0000-0003-2132-0989)
- Yuyue Jiao (ORCID: https://orcid.org/0009-0000-2297-594X)
- Ting ZHANG (ORCID: https://orcid.org/0009-0006-8706-3793)
- Wan-qiong Wang
- Jia Li
- Bo Wang (ORCID: https://orcid.org/0009-0005-0904-2068)
Institutions
- Chinese Academy of Sciences (CN)
- Xinjiang Astronomical Observatory (CN)
- University of Chinese Academy of Sciences (CN)
Publication Details
- Journal
- Astronomy and Computing
- Published
- 2026-11-16
- DOI
- https://doi.org/10.1016/j.ascom.2026.101219
- Primary Topic
- Distributed and Parallel Computing Systems
- Type
- article
- Field-Weighted Citation Impact
- 0.00