Enhancing DNS-over-HTTPS Traffic Classification in Heterogeneous Networks Through Latent Space Analysis with a Tabular-Variational Autoencoder and Self-Attention Classifier Model

Cybersecurity threats and attacks are increasing day by day, bringing real focus on Domain Name System (DNS)–based data exfiltration—a stealth technique used by attackers to steal sensitive information from compromised networks. DNS query exchange is the initial part of any data exchange in the Internet and is the most neglected in traditional monitoring systems. These enable attackers to create covert channels to carry out various advanced persistent threats and unauthorized exfiltration attempts. In this research study, we present a novel detection approach of these DNS patterns through low-dimensional latent representations extracted via a Tabular-Variational AutoEncoder (Tab-VAE), specifically tailored for DNS-over-HTTPS (DoH) traffic. The latent space obtained by the Tab-VAE is subsequently fed into a multi-head self-attention classifier to perform a multi-class classification. We evaluated our experiments using the BCCC-CIC-Bell-DNS-2024 dataset, which provides a realistic snapshot of DoH traffic patterns. Notably, the proposed model demonstrated robust generalization across varying batch sizes and achieved competitive performance metrics with an improved accuracy of 80% and precision score of 75% for a batch size of 128. These findings highlight the potential of advanced machine learning architectures in reinforcing cybersecurity posture. By integrating such techniques, organizations can improve the detection of covert DNS-based attacks and better protect sensitive assets against emerging threats. Received: 28 February 2025 | Revised: 4 July 2025 | Accepted: 22 July 2025 Conflicts of Interest The authors declare that they have no conflicts of interest to this work. Data Availability Statement The data supporting the findings of this study are openly available in Behaviour-Centric Cybersecurity Center (BCCC) at https://www.yorku.ca/research/bccc/ucs-technical/cybersecurity-datasets-cds/. Author Contribution Statement Ravi Veerabhadrappa: Conceptualization, Methodology, Software, Formal analysis, Investigation, Resources, Data curation, Writing – original draft, Visualization, Project administration. Poornima Athikatte Sampigerayappa: Validation, Writing – review & editing, Supervision, Project administration.