TCP‑State‑Driven Multilayer Heterogeneous Graph for P2P Malicious Traffic Detection
Abstract In decentralized peer-to-peer (P2P) networks, distributed denial-of-service (DDoS) attacks can resemble normal multi-source concurrent communication, making detection challenging. Although DDoS attacks can exploit different transport protocols, TCP-based attacks remain an important threat because they can target connection establishment and state maintenance, exhausting connection-related resources. However, existing detection methods often aggregate packets across communication stages into session-level statistics, overlooking TCP state transitions and making it difficult to distinguish normal interactions from anomalous behavior. To address this issue, this paper proposes a TCP-state-driven multilayer heterogeneous graph method for malicious traffic detection in P2P environments. First, a multilayer heterogeneous graph comprising packet nodes and client and server state nodes is constructed based on the TCP finite state machine. Temporal, state transition, and mapping edges capture relationships between packets and communication states. Second, packets are divided into state-specific subsequences, from which statistical features are extracted to characterize fine-grained communication anomalies. Experiments on the CTU-13, CICIDS2018, and ISCXVPN2016 datasets achieve a maximum F1 score of 99.998\%. Representation effectiveness experiments further demonstrate that state-level features distinguish malicious from benign traffic more effectively than whole-session features. These results demonstrate the method's effectiveness in detecting multiple types of malicious traffic and its potential for fine-grained anomaly detection in distributed environments such as P2P networks.
Authors
- Minyu Jin
- Xinlong Wu
- Dexian Zhu
- Bin Li
- Xun Gong
- Jiachen Guo
Institutions
- Sichuan University (CN)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-13
- DOI
- https://doi.org/10.21203/rs.3.rs-11386616/v1
- Primary Topic
- Network Security and Intrusion Detection
- Type
- preprint