Neural Hidden Markov Models for Semi-Supervised Detection of Brute Force Attacks

Cybersecurity threats such as brute force attacks against Secure Shell (SSH) and File Transfer Protocol (FTP) services are increasingly sophisticated, making them challenging to detect using static rules or signature-based techniques. This study investigates a Neural Hidden Markov Model (Neural HMM) approach to identify such attacks within a semi-supervised, one-class learning framework using the widely recognized CICIDS2018 dataset. Unlike traditional methods that rely on extensive labeled attack data, our proposed model is trained exclusively on benign network traffic, thereby learning the probabilistic representation of normal behavior. We embed a Long Short-Term Memory (LSTM)-based neural network within the HMM emission model to capture complex temporal dependencies and non-linear patterns present in high-dimensional network streams. Experimental results demonstrate that the Neural HMM significantly outperforms numerous traditional machine learning and deep learning baselines in terms of F1-score, precision, and recall. Notably, it achieves near state-of-the-art performance while requiring far fewer labeled attack samples. These findings highlight the potential of Neural HMMs as robust, scalable, and adaptive solutions for modern intrusion detection, particularly in dynamic environments with limited labeled data and evolving attack strategies.