DGA-Based Malware Communication Detection from DoH Traffic Using Hierarchical Machine Learning Analysis

Encrypted domain name resolution is increasingly being used to protect the privacy of Internet users, but it may prevent network administrators from detecting malicious communications. Unfortunately, DGA-based malware can exploit it to hide the domain names it generates, so network administrators need a monitoring framework to maintain network security. In this paper, we propose a novel malware detection system using hierarchical machine learning analysis, which incorporates machine learning models, including XGBoost, LightGBM, CatBoost, and RGF. The evaluation results confirm that the proposed system can detect DGA-based malware communication generated by PadCrypt, Sisron, Tinba, and Zloader with 99.19% accuracy. The results show that the proposed system can detect DGA-based malware communications from DoH traffic with sufficient accuracy to support network administrators.