Topology-aware unsupervised intrusion detection from network flows in encrypted traffic

The rapid growth of encrypted network traffic challenges intrusion detection systems that rely on payload inspection. We propose a topology-aware unsupervised intrusion detection framework that models network flows as interaction graphs and applies a graph neural autoencoder to identify structural anomalies from flow metadata alone, without inspecting packet payloads. The autoencoder is trained with a topology-aware augmentation strategy (Gaussian feature noise, edge feature masking, node feature masking) together with edge-level contrastive learning. We evaluate the framework on six datasets organized into eight conditions, spanning enterprise, cloud, industrial IoT and testbed networks as well as end-to-end encrypted DNS-over-HTTPS (DoH) tunneling and command-and-control (C2) traffic over TLS and QUIC. Against six unsupervised baselines, including GraphIDS, Anomal-E, KitNET, and a variational autoencoder, the framework attains the highest mean ROC-AUC on both the general benchmarks (0.986) and the encrypted conditions (0.936). A benign-vs-benign origin control reveals a dataset-origin confound in cross-corpus DoH evaluation (origin-AUC 0.942), so the encrypted-DoH claim rests on a same-origin setting, where the proposed model reaches ROC-AUC 0.797. Without recalibration on the target, the model loses 3.0% (relative) of its mean ROC-AUC under cross-environment transfer and attains a mean cross-domain ROC-AUC of 0.911, compared with 0.415–0.609 for the baselines, and it processes a two-minute window in under 18 ms on a CPU, the highest throughput among the timed models.

Authors

Institutions

Publication Details

Journal
Computer Networks
Published
2026-11-17
DOI
https://doi.org/10.1016/j.comnet.2026.112940
Primary Topic
Network Security and Intrusion Detection
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Topology-aware unsupervised intrusion detection from network flows in encrypted traffic

Taejin Lee, Jung Hye Ran
3 links
Computer Networks
Network Security and Intrusion Detection
article

Topology-aware unsupervised intrusion detection from network flows in encrypted traffic

Taejin Lee, Jung Hye Ran
article en

Abstract

The rapid growth of encrypted network traffic challenges intrusion detection systems that rely on payload inspection. We propose a topology-aware unsupervised intrusion detection framework that models network flows as interaction graphs and applies a graph neural autoencoder to identify structural anomalies from flow metadata alone, without inspecting packet payloads. The autoencoder is trained with a topology-aware augmentation strategy (Gaussian feature noise, edge feature masking, node feature masking) together with edge-level contrastive learning. We evaluate the framework on six datasets organized into eight conditions, spanning enterprise, cloud, industrial IoT and testbed networks as well as end-to-end encrypted DNS-over-HTTPS (DoH) tunneling and command-and-control (C2) traffic over TLS and QUIC. Against six unsupervised baselines, including GraphIDS, Anomal-E, KitNET, and a variational autoencoder, the framework attains the highest mean ROC-AUC on both the general benchmarks (0.986) and the encrypted conditions (0.936). A benign-vs-benign origin control reveals a dataset-origin confound in cross-corpus DoH evaluation (origin-AUC 0.942), so the encrypted-DoH claim rests on a same-origin setting, where the proposed model reaches ROC-AUC 0.797. Without recalibration on the target, the model loses 3.0% (relative) of its mean ROC-AUC under cross-environment transfer and attains a mean cross-domain ROC-AUC of 0.911, compared with 0.415–0.609 for the baselines, and it processes a two-minute window in under 18 ms on a CPU, the highest throughput among the timed models.

Computer NetworksVol. 292
Gachon University (KR)
Gachon University, Ministry of Science and ICT, South Korea, Institute for Information and Communications Technology Promotion
Network Security and Intrusion Detection
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon