An Agentic AI Framework for Autonomous Cross-View Kernel Rootkit Detection and Response in Cloud Environments

Abstract Kernel-level rootkits subvert the operating system to conceal the network ports through which they maintain covert channels, and cloud tenants who cannot access the hypervisor have few reliable defences. Existing detectors stop at a verdict, leaving investigation and response to human analysts and admitting a window during which an active rootkit persists. This paper presents an agentic AI framework that both detects and autonomously responds to kernel rootkits through port-level behavioural analysis. On cloud guests hosted on Google Cloud Platform, a kernel-mode Port Enumerator and a user-mode Port Monitor produce cross-view port lists whose discrepancy, formalized here as a temporal behavioural signal, is classified by a deep ensemble of CNN-LSTM, Bi-LSTM, Autoencoder and GAN. The resulting Rootkit/Benign verdict is handed to an LLM Agent Orchestrator that executes an autonomous perceive-plan-act-reflect loop, invoking security tools under human-in-the-loop oversight for high-impact actions. A second evidence channel is released with the paper as a labelled concealment matrix of 87 kernel rootkit samples described by a binary vector over a 968-entry dictionary of concealed objects, each carrying the technique it uses to hide. We give a mathematical model of the cross-view signal, the fused decision, and the agent’s utility-driven action selection. Evaluated on a Windows Server 2025 guest, the framework detects rootkits with a mean detection latency of 1.2 s while autonomously triaging confirmed threats; on the released concealment matrix the technique of a confirmed rootkit is recovered from its hook vector at 0.846 accuracy against a 0.713 majority-class rate, with the three-way distinction among inline, SSDT and IAT concealment still limited by class imbalance. The response layer is specified and instantiated but not yet measured; Section 5.6 states the protocol by which it will be tested.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-13
DOI
https://doi.org/10.21203/rs.3.rs-10762752/v1
Primary Topic
Security and Verification in Computing
Type
preprint

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

An Agentic AI Framework for Autonomous Cross-View Kernel Rootkit Detection and Response in Cloud Environments

Sudalai Muthu T., Suresh Kumar S
2 links
Research Square
Security and Verification in Computing
preprint

An Agentic AI Framework for Autonomous Cross-View Kernel Rootkit Detection and Response in Cloud Environments

Sudalai Muthu T., Suresh Kumar S
preprint en

Abstract

Abstract Kernel-level rootkits subvert the operating system to conceal the network ports through which they maintain covert channels, and cloud tenants who cannot access the hypervisor have few reliable defences. Existing detectors stop at a verdict, leaving investigation and response to human analysts and admitting a window during which an active rootkit persists. This paper presents an agentic AI framework that both detects and autonomously responds to kernel rootkits through port-level behavioural analysis. On cloud guests hosted on Google Cloud Platform, a kernel-mode Port Enumerator and a user-mode Port Monitor produce cross-view port lists whose discrepancy, formalized here as a temporal behavioural signal, is classified by a deep ensemble of CNN-LSTM, Bi-LSTM, Autoencoder and GAN. The resulting Rootkit/Benign verdict is handed to an LLM Agent Orchestrator that executes an autonomous perceive-plan-act-reflect loop, invoking security tools under human-in-the-loop oversight for high-impact actions. A second evidence channel is released with the paper as a labelled concealment matrix of 87 kernel rootkit samples described by a binary vector over a 968-entry dictionary of concealed objects, each carrying the technique it uses to hide. We give a mathematical model of the cross-view signal, the fused decision, and the agent’s utility-driven action selection. Evaluated on a Windows Server 2025 guest, the framework detects rootkits with a mean detection latency of 1.2 s while autonomously triaging confirmed threats; on the released concealment matrix the technique of a confirmed rootkit is recovered from its hook vector at 0.846 accuracy against a 0.713 majority-class rate, with the three-way distinction among inline, SSDT and IAT concealment still limited by class imbalance. The response layer is specified and instantiated but not yet measured; Section 5.6 states the protocol by which it will be tested.

Research Square
Hindustan Institute of Technology and Science (IN), Rajalakshmi Engineering College (IN)
Department of Artificial Intelligence, Korea University
Security and Verification in Computing
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon