Topology-aware unsupervised intrusion detection from network flows in encrypted traffic
The rapid growth of encrypted network traffic challenges intrusion detection systems that rely on payload inspection. We propose a topology-aware unsupervised intrusion detection framework that models network flows as interaction graphs and applies a graph neural autoencoder to identify structural anomalies from flow metadata alone, without inspecting packet payloads. The autoencoder is trained with a topology-aware augmentation strategy (Gaussian feature noise, edge feature masking, node feature masking) together with edge-level contrastive learning. We evaluate the framework on six datasets organized into eight conditions, spanning enterprise, cloud, industrial IoT and testbed networks as well as end-to-end encrypted DNS-over-HTTPS (DoH) tunneling and command-and-control (C2) traffic over TLS and QUIC. Against six unsupervised baselines, including GraphIDS, Anomal-E, KitNET, and a variational autoencoder, the framework attains the highest mean ROC-AUC on both the general benchmarks (0.986) and the encrypted conditions (0.936). A benign-vs-benign origin control reveals a dataset-origin confound in cross-corpus DoH evaluation (origin-AUC 0.942), so the encrypted-DoH claim rests on a same-origin setting, where the proposed model reaches ROC-AUC 0.797. Without recalibration on the target, the model loses 3.0% (relative) of its mean ROC-AUC under cross-environment transfer and attains a mean cross-domain ROC-AUC of 0.911, compared with 0.415–0.609 for the baselines, and it processes a two-minute window in under 18 ms on a CPU, the highest throughput among the timed models.
Authors
- Taejin Lee (ORCID: https://orcid.org/0000-0003-3078-3459)
- Jung Hye Ran (ORCID: https://orcid.org/0009-0007-0697-7639)
Institutions
- Gachon University (KR)
Publication Details
- Journal
- Computer Networks
- Published
- 2026-11-17
- DOI
- https://doi.org/10.1016/j.comnet.2026.112940
- Primary Topic
- Network Security and Intrusion Detection
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- Gachon University
- Ministry of Science and ICT, South Korea
- Institute for Information and Communications Technology Promotion