Unsupervised Malicious Domain Detection Using Auto-Encoder Models with Reconstruction Error

Domain Generation Algorithms (DGAs) are commonly used by malware to generate large numbers of pseudo-random domain names for command-and-control (C2) communication, making detection increasingly difficult. Traditional supervised approaches rely heavily on labeled datasets that include malicious domains, which are costly to obtain and often fail to generalize to unseen DGAs. In this paper, we propose an unsupervised DGA detection method based on Auto-Encoder (AE) and Variational Auto-Encoders (VAE) trained solely on benign domains and leverages reconstruction error to detect DGA-based anomalies without prior exposure to malicious labels. Experimental results benchmarked on Cisco Umbrella Top 1 Million datasets and several different DGA families show that the proposed VAE model achieves superior detection performance with an AUC of 0.83 compared with other unsupervised baselines. This demonstrates the effectiveness of one-class learning frameworks in generalizing to previously unseen threats, providing a scalable and low-maintenance solution for real-world domain anomaly detection.