On-the-fly data reduction by streaming processing of visibilities

Realising the scientific ambitions of ALMA's Wideband Sensitivity Upgrade (WSU) will require a significant increase in IF and correlator bandwidth. This will result in a larger output data rate of the correlator, putting increasing demands on storage capacity and I/O for the buffering of raw visibility data. To reduce the costs (in terms of energy consumption and hardware) of handling the raw visibility data, we started the Streaming Visibility Processing (SVP) development study in September 2023. The aim of this project is to demonstrate a prototype data processing pipeline that captures the visibility output stream from the ALMA correlator, performs a number of observation-dependent visibility processing steps and only stores the visibilities after reducing the visibility data volume. Possible examples of such data-volume reducing initial processing steps are Baseline-Dependent Averaging (BDA) and visibility data compression using the Dysco algorithm. Although these techniques are not lossless, they can be configured in such a way that the increase in the thermal noise level of the final images is less than 1\% while the data volume is reduced by an order of magnitude. A demonstration of this trade-off and its implications for ALMA science is part of the development study.