Comparative Analysis of Architectures for Classifying Malicious and Benign TLS Traffic

The increasing prevalence of encrypted internet traffic has presented significant challenges in detecting and classifying malicious activities. Transport Layer Security (TLS), a protocol for secure communications, is widely adopted but has created blind spots for traditional cybersecurity tools that rely on traffic content inspection. This study addresses the challenge by evaluating the use of three machine learning models—a random forest, a long short-term memory network (LSTM), and a fully connected deep neural network (FDCNN)—to classify TLS traffic into benign and malicious categories. Using the CIC-IDS-2017 dataset, these models leverage features such as packet size, timing, and connection frequency, enabling classification without decrypting the traffic. Experimental results demonstrate that the random forest model achieves the highest accuracy (99.02%), followed by the FDCNN (98.75%) and the LSTM (98.00%). Each model shows strengths in specific areas, with the random forest excelling in handling imbalanced classes, the FDCNN capturing complex patterns, and the LSTM identifying temporal dependencies. These findings contribute to advancing TLS traffic classification and highlight the potential of machine learning in addressing encrypted traffic analysis challenges.