TCP‑State‑Driven Multilayer Heterogeneous Graph for P2P Malicious Traffic Detection

Abstract In decentralized peer-to-peer (P2P) networks, distributed denial-of-service (DDoS) attacks can resemble normal multi-source concurrent communication, making detection challenging. Although DDoS attacks can exploit different transport protocols, TCP-based attacks remain an important threat because they can target connection establishment and state maintenance, exhausting connection-related resources. However, existing detection methods often aggregate packets across communication stages into session-level statistics, overlooking TCP state transitions and making it difficult to distinguish normal interactions from anomalous behavior. To address this issue, this paper proposes a TCP-state-driven multilayer heterogeneous graph method for malicious traffic detection in P2P environments. First, a multilayer heterogeneous graph comprising packet nodes and client and server state nodes is constructed based on the TCP finite state machine. Temporal, state transition, and mapping edges capture relationships between packets and communication states. Second, packets are divided into state-specific subsequences, from which statistical features are extracted to characterize fine-grained communication anomalies. Experiments on the CTU-13, CICIDS2018, and ISCXVPN2016 datasets achieve a maximum F1 score of 99.998\%. Representation effectiveness experiments further demonstrate that state-level features distinguish malicious from benign traffic more effectively than whole-session features. These results demonstrate the method's effectiveness in detecting multiple types of malicious traffic and its potential for fine-grained anomaly detection in distributed environments such as P2P networks.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-13
DOI
https://doi.org/10.21203/rs.3.rs-11386616/v1
Primary Topic
Network Security and Intrusion Detection
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

TCP‑State‑Driven Multilayer Heterogeneous Graph for P2P Malicious Traffic Detection

Minyu Jin, Xinlong Wu, Dexian Zhu, Bin Li et al.
1 links
Research Square
Network Security and Intrusion Detection
preprint

TCP‑State‑Driven Multilayer Heterogeneous Graph for P2P Malicious Traffic Detection

Minyu Jin, Xinlong Wu, Dexian Zhu, Bin Li, Xun Gong, Jiachen Guo
preprint en

Abstract

Abstract In decentralized peer-to-peer (P2P) networks, distributed denial-of-service (DDoS) attacks can resemble normal multi-source concurrent communication, making detection challenging. Although DDoS attacks can exploit different transport protocols, TCP-based attacks remain an important threat because they can target connection establishment and state maintenance, exhausting connection-related resources. However, existing detection methods often aggregate packets across communication stages into session-level statistics, overlooking TCP state transitions and making it difficult to distinguish normal interactions from anomalous behavior. To address this issue, this paper proposes a TCP-state-driven multilayer heterogeneous graph method for malicious traffic detection in P2P environments. First, a multilayer heterogeneous graph comprising packet nodes and client and server state nodes is constructed based on the TCP finite state machine. Temporal, state transition, and mapping edges capture relationships between packets and communication states. Second, packets are divided into state-specific subsequences, from which statistical features are extracted to characterize fine-grained communication anomalies. Experiments on the CTU-13, CICIDS2018, and ISCXVPN2016 datasets achieve a maximum F1 score of 99.998\%. Representation effectiveness experiments further demonstrate that state-level features distinguish malicious from benign traffic more effectively than whole-session features. These results demonstrate the method's effectiveness in detecting multiple types of malicious traffic and its potential for fine-grained anomaly detection in distributed environments such as P2P networks.

Research Square
Sichuan University (CN)
Network Security and Intrusion Detection
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon