MambaGCN-SMI: Protocol state machine inference framework based on Mamba and GCN

To address the limitations of existing non-interactive protocol state machine inference methods in feature representation capability, inter-stage information fusion, and adaptive simplification strategies, this paper proposes the MambaGCN-SMI inference framework. The method employs a dual-level Mamba encoder to extract deep semantic features from protocol interactions, constructs an initial directed state graph based on clustering and state transition conditional entropy, and utilizes a Graph Convolutional Network to learn node embeddings, enabling data-driven adaptive state merging and simplification. Experiments on a multi-protocol dataset encompassing TCP, HTTP, SMTP, Modbus, and DNP3 demonstrate that the framework outperforms mainstream methods in both state machine structure recovery and behavioral prediction. It achieves an average prediction accuracy of 95.7% and a behavior coverage of 97.5%. For structural recovery, the node and edge F1-scores reach 0.939 and 0.915, respectively. Ablation studies confirm that both the Mamba encoder and the Graph Convolutional Network module contribute significantly to the performance improvement. By integrating selective state space modeling with graph structure learning, this research achieves high-precision, adaptive inference of private protocol state machines, offering an effective deep learning solution for protocol reverse engineering analysis.