Data Pipeline System Designs for In-network Learning

This paper introduces the design of a data pipeline system (DPS) integrated with artificial intelligence (AIF) functions to support continuous AI learning and operations for network automation in 5G/6G systems. We design the DPS as a chain of functions, namely ingress and egress Network Data Broker Function (iNDBF and eNDBF) and Network Data Preprocessing Function (NDPPF), to support in-network learning operations. To take into account the distributed nature of the network architecture of 5G systems and beyond, we conceive the DPS to be integrated seamlessly with a distributed learning frameworks such as the federated learning (FL). We performed a realistic evaluation, employing a real dataset from a national mobile operator to simulate the network architecture. Additionally, a FL framework for anomaly detection is integrated with the DPS to assess the effectiveness of our proposal. Evaluation results show that delays in end-to-end data transmission and preprocessing to the AIF locations can cause distributed learning AIFs to work with stale data. The results also highlight how the DPS can counterbalance these delays leading to desynchronisation of the distributed learning process, bringing to AIFs with higher accuracy.