Network-based anomaly detection in encrypted data streams : A cryptanalysis perspective

Encrypted traffic now comprises most Internet flows, rendering traditional inspection methods ineffective and posing significant challenges for real-time anomaly detection. In this work, we introduce a layered detection framework that combines (1) a cryptanalysis-informed scoring function quantifying deviations in flow entropy and handshake parameter statistics with (2) a hybrid decision pipeline that employs a lightweight decision tree to filter benign traffic before invoking a secondary SVM/Random-Forest classifier, and (3) an adaptive thresholding and retraining strategy based on exponentially weighted moving averages and daily model updates. Evaluated on real-world TLS 1.3 captures, the CTU-13 botnet dataset, and instrumented QUIC traces, our approach achieves a 92 % detection rate and a 2 % false-positive rate, while processing each flow in under 1 ms. These results demonstrate that fusing domain-specific cryptanalytic insights with adaptive machine learning yields both high accuracy and operational efficiency for encrypted-traffic monitoring.