Lurking in the shadows: Unsupervised decoding of beaconing communication for enhanced cyber threat hunting

The escalating prevalence of Advanced Persistent Threats (APTs) necessitates the development of more robust solutions capable of effectively thwarting these attacks by monitoring system activities across individual hosts. Existing cloud-native security applications utilize a combination of rule-based and machine learning-based detection techniques to protect digital assets. However, these approaches have limitations. Rule-based detection depends on predefined rules to identify specific attack patterns. Persistent attackers can often evade detection by carefully ensuring that their behavior circumvents these rules. In contrast, machine learning-based detection techniques, which learn attack patterns from data, rely heavily on the availability of labeled data for training. However, labeled data is often unavailable and can be labor-intensive and costly to obtain. In this paper, we address the challenge of detecting APT attacks more holistically by leveraging attackers’ behavior during communication with Command and Control (C2) servers, a critical phase observed in most APT attacks. We aim to reduce false positive alerts for threat hunters by analyzing system network logs to detect potential network beaconing, a common attribute of various malware. We introduce a novel hybrid approach, called NetSpectra Sentinel, which employs a Continuous Time Hidden Markov Model (CT-HMM) to detect hidden states underlying observed patterns within the network logs and Time Series Decomposition (TSD) to model temporal patterns. We evaluate the effectiveness of our approach using 14 benchmark datasets and one synthetic dataset, comparing our method with other state-of-the-art statistical-based and botnet detection techniques. The results demonstrate that our technique achieves significantly higher accuracy in most cases, and even when existing techniques fail, our approach can still detect beaconing post-initial compromise with up to 90% accuracy. Additionally, we achieve up to four times better performance in terms of precision compared to existing statistical-based techniques.