Performance vs. Accuracy Trade-Off Analysis of Tree-Based Machine Learning and Deep Learning Models for Real-Time Network Intrusion Detection

Background & Problem Statement: Modern enterprise networks operate across rapidly expandingattack surfaces, driving the transition from static signature-based Network Intrusion Detection Systems(NIDS) to Machine Learning (ML) and Deep Learning (DL) flow classifiers. Although deep neuralnetworks and large gradient-boosted ensembles achieve near-perfect offline classification scores, theirhigh computational complexity and memory footprints risk inducing severe processing bottlenecks andpacket drops under strict multi-gigabit line-rate Service Level Agreements (< 1.0 ms per flow).Methodology: This thesis introduces the Unified Dual-Fidelity Benchmarking and Pareto-OptimizationFramework (UDBF-NIDS) evaluated on the 2,830,743-flow CICIDS2017 dataset. A leakage-free preprocessing pipeline strips static socket identifiers, enforces a 70/15/15 stratified split, applies trainingfitted Robust Scaling, selects 34 non-collinear features via Spearman rank correlation and Tree-SHAPattribution, and mitigates class imbalance via SMOTE+Tomek Links, class-balanced weighting, and Focal Loss. Four tree-based estimators (CART Decision Tree, Random Forest, XGBoost, LightGBM) andtwo DL architectures (MLP, 1D-CNN) are benchmarked across 424,152 holdout test flows spanning 15classes on a single-threaded Intel Core i5-8265U CPU testbed using nanosecond performance countersand micro-batched (B = 1,024) throughput profiling.Key Empirical Findings: Tree-based models decisively outperformed DL architectures across overalland minority attack categories due to the inductive bias mismatch of continuous neural projections onheterogeneous tabular telemetry. XGBoost achieved the highest detection fidelity with 0.998654 accuracy, 0.998649 weighted F1-score, and 0.862091 macro F1-score—including top minority F1-scoreson Infiltration (0.500000) and Heartbleed (1.000000)—but incurred a 3.035580 ms (3,035.58 µs) meansingle-sample latency and 9,966.58 samples/s batch throughput due to CPU L3 cache evictions across4,500 trees (10.49 MB). Conversely, MLP recorded the lowest mean latency at 0.116416 ms (116.42 µs)and 1D-CNN achieved 0.297770 ms (297.77 µs), yet both failed operationally with weighted F1-scoresof 0.764465 and 0.737203 (macro F1 of 0.286097 and 0.147556). The cache-resident CART DecisionTree (0.46 MB) achieved a 0.997430 weighted F1-score, 0.792267 macro F1-score, the highest balancedaccuracy (0.848016), a 0.125687 ms (125.69 µs) mean latency (0.236430 ms P95, 0.4170 ms maximum),and the highest batch throughput at 2,698,884.72 samples/s.Conclusion: Multi-objective Pareto and queueing-theoretical analyses eliminate RandomForest(10.287557ms, 0.996382 weighted F1), LightGBM (1.198944 ms, 0.344702 weighted F1), and 1D-CNN as strictlydominated. Because transitioning from Decision Tree to XGBoost yields only a +0.001219 weighted F1gain at a 24.15-fold latency penalty and 270.79-fold throughput reduction, the single Decision Tree represents the sole Pareto-optimal architecture capable of sustaining deterministic < 1.0 ms inline line-rateinspection without ingress queue saturation