Diving into large-scale congestion with NOTED as a network controller and machine learning-based traffic forecasting

The Network Optimised Transfer of Experimental Data (NOTED) has undergone successful testing at several international conferences, including the International Conference for High Performance Computing, Networking, Storage and Analysis (also known as SuperComputing). It has also been tested at scale during the WLCG Data Challenge 2024 (DC24), in which NREN and WLCG sites conducted testing at 25% of the rates foreseen for the High-Luminosity LHC (HL-LHC). During these events, NOTED has demonstrated its ability to detect network congestion and dynamically reconfigure the network by executing actions, thereby enhancing network utilisation. Recently, the integration of NOTED with the CERN’s Network Monitoring System has increased its ability to detect and respond to congestion in the LHCOPN (Tier 0 to Tier 1’s links) and LHCONE (Tier 1’s to Tier 2’s links) networks. We report here on NOTED’s enhanced ability to identify congested WLCG sites and DC24 experiences with network reconfiguration to alleviate the detected congestion. Previous work has demonstrated the feasibility of improving NOTED’s ability to predict network traffic using machine learning with LSTM (Long Short-Term Memory) networks, given its capacity to learn from historical data. We present here new findings on the beneficial impact of various encoderdecoder-based machine learning architectures, including sequence-to-sequence (Seq2Seq) models, Autoencoders, and Transformers, on NOTED’s performance in relation to traffic forecasting.