Optimizing decision trees for multi-class VPN and non-VPN traffic classification using BBFS-PSO

Accurate classification of Virtual Private Network (VPN) and non-VPN network traffic is critical for effective management and security in today’s encrypted and privacy-conscious network environments. This study introduces a multi-class traffic classification pipeline that first extracts comprehensive flow-level statistical features from raw packet captures using CICFlowMeter. Decision Tree models are then employed as the core classifier, enhanced by Bowerbird Courtship-Inspired Feature Selection (BBFS) for optimal feature reduction and Particle Swarm Optimization (PSO) for hyperparameter tuning. The proposed framework is evaluated on three benchmark datasets—ISCXVPN2016, the VPN/Non-VPN Network Application Traffic Dataset (VNAT) and the Encrypted VPN Dataset—encompassing a range of application types, including streaming, VoIP, and remote desktop, under both VPN and non-VPN scenarios. Experimental results show that the BBFS algorithm effectively selects highly compact and informative feature subsets from the CICFlowMeter-extracted data, while PSO optimization yields lightweight and interpretable Decision Trees. To demonstrate model-agnostic utility, we additionally instantiate the pipeline with a Random Forest classifier, observing parallel gains in F1 and efficiency that confirm transferability beyond a single learner. The pipeline consistently outperforms traditional methods, delivering robust classification accuracy, significant reductions in tree depth and complexity, and stable performance across various traffic classes and intervals. These results establish the practicality and generalizability of the CICFlowMeter-based BBFS-PSO-optimized Decision Tree approach for real-world encrypted traffic analysis, paving the way for efficient and transparent network monitoring solutions.