IHPRE: An improved hierarchical protocol reverse engineering approach

Currently, numerous private protocols operate on Internet and Internet of Things devices, yet their specifications remain largely unknown. Many network security facilities depend on detailed protocol information, thereby necessitating the reverse parsing of protocol data. A common approach is analyzing network traffic traces from protocol interactions to infer message format. However, existing research has a latent assumption: it implicitly assumes messages comprise only headers and payloads, overlooking trailers, which distorts format inference. Furthermore, these methods handle message headers and message payloads uniformly during message clustering, overlooking the disparity in information volume between them. Consequently, the clustering reveals a substantial quantity of scattered and disorderly data, accompanied by an unduly high proportion of noise. To tackle these issues, this paper proposes a novel hierarchical protocol reverse engineering method that iteratively traverses the protocol's hierarchical structure. In each layer, information entropy first identifies the most probable protocol boundaries, segmenting the protocol into three parts. Then, customized analysis is performed on each part. Evaluation using seven widely adopted protocols shows the method's superiority in field inference.