INGAD: Asynchronous acknowledgment channel decoupling for high-throughput data archiving in radio astronomy

The Qitai Radio Telescope (QTT), a 110 m fully steerable dish under construction in China, will produce extremely large volumes of observational data that must be transferred efficiently and reliably from the observatory site to remote data center. The Next Generation Archive System (NGAS) is a widely used archival system in astronomy, but it suffers from stop-and-wait flow control, Global Interpreter Lock (GIL) bottlenecks, and a lack of backpressure, which limit its throughput and scalability for next-generation facilities. We present INGAD (Improved NGAS based on Asynchronous Acknowledgment Channel Decoupling), a new data transfer architecture that fully decouples the data and acknowledgment paths. The data channel uses ZeroMQ PUSH-PULL sockets for high-speed asynchronous transfer with inherent backpressure, while the acknowledgment channel employs a persistent RabbitMQ message queue to deliver ACKs independently without blocking the sender. A bounded in-flight sliding window limits the number of unacknowledged files, preventing unbounded memory growth. Dynamic timeout estimation based on historical ACK delays, together with exponential backoff retransmission and idempotent deduplication, ensures reliable delivery under variable network conditions. We evaluate INGAD using both synthetic simulation data and real ASKAP observations over a 5 Gbit/s network.Compared to the original NGAS, INGAD improves throughput by 115%–132% for small files and 22%–28% for large and ultra-large files in the simulation campaign (with gains of 5.8%–115% over the coupled ZeroMQ baseline). In the real ASKAP data campaign, throughput gains over NGAS are 22%–27% for small files and 16%–22% for large/ultra-large files, reducing total transfer time by up to 56% for small files and 13% for ultra-large files. Theoretical modeling confirms that INGAD achieves 39%–74.5% of the 5 GbE link limit, with prediction errors below 3.2%. Memory usage remains bounded and predictable across all tests, with no out-of-memory events. INGAD provides a scalable, robust, and high-performance solution for astronomical data transfer, making it particularly well suited for the QTT and other large-scale observatories requiring efficient remote data synchronization.