Enabling Hybrid Edge-Level Threat Detection in O-RAN Private Networks for Resilience Enhancement

As private 4G and 5G networks become increasingly critical for industrial applications such as smart manufacturing and autonomous systems, ensuring the security and reliability of the Radio Access Network (RAN) is essential. Traditional mobile network defenses rely heavily on centralized analysis at the core network, resulting in latency and limited scalability. To address these challenges, the Open Radio Access Network (O-RAN) architecture introduces modular and programmable components that enable security mechanisms to operate closer to the network edge. In this paper, we propose a hybrid detection framework that integrates a rule-based traffic filter at the O-CU with an intelligent xApp deployed on the Near-RT RIC. The system performs GTP decapsulation and flow-level feature extraction at the O-CU, then transmits the data to the xApp, which employs an LSTM-based detector to identify low-rate Denial-of-Service attacks. Detection results are fed back to dynamically update local filtering rules. Experimental results demonstrate that the proposed framework enables lightweight and near real-time threat mitigation, thereby reducing reliance on core-network-based analysis and enhancing the responsiveness of RAN-level defenses.