Enhanced Autoencoder-Based Clustering for Message Analysis in Binary Protocols

Reverse engineering is a critical process in software engineering and communication networks, playing a vital role in optimizing protocols and complex systems. By enabling the analysis and understanding of existing structures, reverse engineering enhances system performance and supports the development of innovative solutions to improve communication efficiency and security. A key aspect of protocol reverse engineering is identifying message types, which is essential for understanding both individual message functions and the overall protocol. Accurate message classification, particularly through clustering algorithms that group messages based on similarities, is crucial for achieving reliable identification results. In this study, after evaluating and comparing various methods, two superior approaches-AEMK and EAEAP-have been identified. These methods integrate two types of autoencoders with Affinity Propagation algorithms and Mini-Batch K-Means. The findings reveal that these proposed approaches significantly enhance cluster homogeneity compared to other methods, such as the PREUNN tool. Specifically, while the PREUNN tool achieves an average homogeneity of 0.193, the AEMK method reaches 0.554, and the EAEAP method achieves 0.919. These results demonstrate that the proposed methods effectively cluster binary protocols, thereby improving the accuracy of message type identification and advancing the field of protocol reverse engineering.