A transport-layer cryptographic framework secures inter-agent communication and verdict provenance in multi-agent malware detection pipelines

Abstract Multi-agent AI systems have emerged as a promising approach for metamorphic malware detection, combining large language model (LLM) reasoning with specialized static, dynamic, and similarity-analysis tools. The cryptographic security of the supporting infrastructure – inter-agent channels, agent identities, and signed analytic verdicts – has so far received less attention than the detection algorithms themselves. This paper presents the Secure Agent Communication Protocol (SACP), a transport-layer cryptographic framework for multi-agent malware detection pipelines. SACP composes standard primitives (X25519, HKDF-SHA256, AES-256-GCM, Ed25519) into a TLS 1.3-style mutually-authenticated handshake adapted to a Certificate-Authority-mediated multi-agent setting, providing confidentiality, integrity, mutual authentication, replay resistance, forward secrecy, and signed-result accountability. We explicitly scope the contribution: SACP secures infrastructure-layer threats and does not address Agentic-AI-specific threats such as prompt injection, indirect prompt injection, tool-call abuse, or model extraction, which we treat as complementary and orthogonal. We evaluate SACP on a corpus of 10,000 metamorphic malware samples and 5000 benign Windows executables (15,000 samples in total; 3000-sample held-out test set), measuring a single-run cryptographic overhead of $$\approx 0.10\%$$ relative to an unsecured agentic baseline while preserving detection performance (F1 0.948 with SACP vs. 0.951 without; accuracy 0.930 vs. 0.934). We provide a qualitative comparison against TLS 1.3, Noise, and MLS, and a discussion of which protocol-level attacks (UKS, KCI, replay, identity misbinding) the construction does and does not directly resist. We close with an explicit limitations section that scopes the experimental evidence and lists the validation work – modern corpora, multi-run variance, mechanized proofs, and red-team attack experiments – that remains for future work.