On-the-fly, memory-aware unsupervised learning for network anomaly detection: A systematic literature review

Modern enterprise networks, 5G/6G, and Internet of Things infrastructures generate massive, heterogeneous, and increasingly encrypted traffic, reducing the operational effectiveness of signature-based intrusion detection. We present a systematic literature review of unsupervised machine learning for network anomaly detection with emphasis on on-the-fly adaptation and memory-aware or memory-prediction mechanisms that can detect previously unseen attacks without labeled data. Following PRISMA 2020 reporting guidance, we screened 2,847 records published from 2015 to 2025 and retained 121 primary studies. Using bibliometric keyword co-occurrence mapping in VOSviewer, we identify four thematic clusters: (i) deep reconstruction and generative models, (ii) edge and next-generation network settings, (iii) privacy-preserving and federated detection, and (iv) deployment-facing concerns including encrypted traffic visibility, explainability, and drift-aware adaptation. Across the corpus, fewer than 15% of studies evaluate genuine streaming updates, explicit cortical-inspired memory-prediction mechanisms appear in only three studies, and public datasets provide limited coverage of encrypted traffic and 5G slicing telemetry. Based on these gaps, we outline AMNAD (Adaptive Memory-Augmented Network Anomaly Detector), a two-stage reference architecture that couples lightweight edge screening with a prototype-memory autoencoder core and an explanation layer to support low-latency, drift-aware anomaly detection under resource constraints. To support reproducibility, we provide the curated reference database and the VOSviewer map and network files used to generate the keyword co-occurrence analysis. • SLR of 121 studies (2015 to 2025) on unsupervised, streaming, and memory-aware network intrusion detection systems (NIDS). • Deep autoencoders dominate, and only ∼ 15% test online or real-time adaptation. • Encrypted QUIC/TLS and 5G slicing are rarely covered by public datasets. • AMNAD: two-stage edge plus prototype-memory autoencoder core with interpretable alerts. • We provide a curated BibTeX reference database and reproducible VOSviewer map/network files.