AI-Driven Network Intrusion Detection and Resource Allocation in Real-World O-RAN 5G Networks

5G technology, the latest advancement in mobile networks, promises increased data speeds, reduced latency, and enhanced capacity. However, network performance and user experience can be critically impacted by malicious traffic, identified as anomaly traffic and intrusion methods. Addressing these challenges requires optimized resource sharing and robust network security measures. In this paper, we present an AI/ML-driven Network Intrusion Detection framework with dynamic resource allocation and user management within the O-RAN architecture. Our Anomaly Traffic Detector (ATD) enhances network security by mitigating Denial of Service (DoS) attacks through an xApp that classifies network traffic in real-time and dynamically adjusts network resources and user connections. Experimental evaluations show that our system effectively maintains low latency under attack conditions, nearly doubles the throughput for legitimate users, and reduces average CPU usage by up to 15%. We use as reference platforms the OpenAirInterface, and FlexRIC for programming the slice and user connectivity decisions at the RAN level, and evaluate our scheme under real-world settings in a testbed environment.