Game-Theoretic Secure Socket Transmission with a Zero Trust Model

A significant problem in cybersecurity is to accurately detect malicious network activities in real-time by analyzing patterns in socket-level packet transmissions. This challenge involves distinguishing between legitimate and adversarial behaviors while optimizing detection strategies to minimize false alarms and resource costs under intelligent, adaptive attacks. This paper presents a comprehensive framework for network security by modeling socket-level packet transmissions and extracting key features for temporal analysis. A long short-term memory (LSTM)-based anomaly detection system predicts normal traffic behavior and identifies significant deviations as potential cyber threats. Integrating this with a zero trust signaling game, the model updates beliefs about agent legitimacy based on observed signals and anomaly scores. The interaction between defender and attacker is formulated as a Stackelberg game, where the defender optimizes detection strategies anticipating attacker responses. This unified approach combines machine learning and game theory to enable robust, adaptive cybersecurity policies that effectively balance detection performance and resource costs in adversarial environments. Two baselines are considered for comparison. The static baseline applies fixed transmission and defense policies, ignoring anomalies and environmental feedback, and thus serves as a control case of non-reactive behavior. In contrast, the adaptive non-strategic baseline introduces simple threshold-based heuristics that adjust to anomaly scores, allowing limited adaptability without strategic reasoning. The proposed fully adaptive Stackelberg strategy outperforms both partial and discrete adaptive baselines, achieving higher robustness across trust thresholds, superior attacker–defender utility trade-offs, and more effective anomaly mitigation under varying strategic conditions.