A Novel Anomaly Detection Framework for Software-Defined Networks Using Bedbug-Optimized Hidden Markov Models

Software-Defined Networks (SDNs) offer programmability and ease of management, but the architectural revolution poses very serious security threats to the conventional intrusion detection system. To overcome the aforementioned problems, this paper proposes Bedbug-HMM, a flow-level anomaly detection framework for SDN controllers that processes NetFlow measurements to identify traffic anomalies causing controller overload. The framework integrates NetFlow data extraction, PCA-enhanced preprocessing, Bedbug Metaheuristic Algorithm (BMA)-optimized Hidden Markov Models with quantile-discretized observation sequences, population-based ensemble scoring, MCC-optimized thresholds, and ECA-based mitigation policies for automated OpenFlow response.The experimental analysis performed on the NSL-KDD and UNSW-NB15 datasets proves the superior performance of the proposed approach over six baselines including clustering, MLP/GNN, Baum-Welch HMM, PSO-HMM, and GA-HMM (with + 5.9% AUC, + 9.1% F1, + 11.2% MCC over MLP/GNN and + 14.8% MCC over Baum-Welch HMM on NSL-KDD), with a resulting 53.5%-63.7% reduction in bandwidth and the ability to process 892-1,072 anomalies per second.