A Time-Series and Density-Based Filter for DNS Log Reduction and Analysis

The Domain Name Service (DNS) is fundamental to the successful operation of the Internet, providing behind the scenes translation between Uniform Resource Locators (URLs) used by humans and machines and the Internet Protocol (IP) addresses required for data transmission between hosts and servers. DNS is ubiquitous across networks and for decades has been used for malicious purposes by threat actors. There has been significant research in detecting DNS protocol abuse leveraging statistical analysis, natural language processing and machine learning. The volume of DNS traffic in enterprise networks is significant and leveraging detection techniques on large datasets is costly in terms of time, processing and memory resources. There is a need to reduce the size of DNS logs to enable more efficient use of detection techniques and reduce the amount of data to be reviewed by analysts. The aim of this research was to develop and evaluate a log filtering technique to reduce DNS log size while retaining sufficient malicious traffic samples to enable efficient analysis and DNS abuse detection. This technique leverages a single time-delta feature and density-based clustering to reduce DNS log size. The results showed up to a 76% decrease in log size by row count and up to 99 % reduction in user IP and DNS query pairs while retaining up to 83% of malicious traffic. Operationally, this provides a much reduced dataset size for analysts that requires less time and computational resources to process.