Optimizing Text-Based Protocol Clustering in Reverse Engineering with Auto-Encoders and Fine-Tuned Parameters

As documentation for unknown protocols is often confidential, reverse engineering becomes essential for analysis. This multi-step process is time-consuming, prompting researchers to automate it using various tools. A key aspect of this process involves classifying similar messages. Traditional clustering methods in deep learning and machine learning are employed. This research aims to evaluate and test clustering methods such as SOM, K-Means, DBSCAN, Agglomerative Clustering (AC), and Affinity Propagation (AP) on text-based protocols. Parameter tuning methods, including auto-encoder feature extraction, enhance cluster homogeneity. Our experiments reveal that combining an auto-encoder with K-Means results in 78% homogeneity, while combining it with DBSCAN achieves 86% homogeneity, making these the most effective approaches for clustering text-based protocols. Specifically, while the PREUNN tool, based on deep learning, has examined two protocols, FTP and HTTP, and achieved an average homogeneity of 77.3% within its dataset, applying this method to our research dataset shows that it only achieves an average homogeneity of 22%. This indicates a significant lack of generalizability; in other words, this method provides acceptable homogeneity only within its specific dataset.