Detecting encrypted malicious traffic with HEAT: a header-focused deep learning approach

Abstract The widespread adoption of encryption in network traffic significantly challenges traditional detection methods that rely on payload analysis. Existing approaches often convert traffic into images or sequences for deep learning models, producing redundant features and struggling with multi-protocol environments. In this study, we propose HEAT (Header-Embedded Attention for Traffic Detection), a novel model that leverages packet header fields to develop a robust characteristic representation for encrypted traffic analysis. HEAT introduces a hierarchical attention mechanism combined with a novel contextual embedding technique that enhances the semantic representation of header field values. Additionally, HEAT integrates an adapted Kolmogorov–Arnold Network classifier with B-spline activations and L1 weight regularization, optimizing the model for efficient real-time processing. Extensive evaluations on CICIDS-2018, Stratosphere, and ISCX2012 datasets demonstrate HEAT’s superior performance, achieving 98.95% accuracy and 98.28% F1-score on CICIDS-2018, 99.5% accuracy and 98.54% F1-score on Stratosphere, and 99.75% accuracy with 99.25% F1-score on ISCX2012. HEAT significantly outperforms CNN, LSTM, and BiGRU baselines. Moreover, it maintains detection accuracy above 98.95% during incremental learning, with only a 0.9% F1-score drop, compared with 6.55% in conventional models. These results highlight HEAT’s novelty, stability, and adaptability, making it a scalable and robust solution for encrypted malicious traffic detection.