Autonomous Penetration Testing with an LLM-Embedded Deep Q-Network: A Proof-of-Concept Study on the NetSecGame Environment

Autonomous penetration testing is a sequential decision problem suited to reinforcement learning (RL). RL agents, however, tend to memorise one network and fail when its layout changes, while planners built on large language models are costly, hard to reproduce, and raise privacy concerns when network data is sent to third parties. We study a middle path in which a language model is the agent's perception, not its decision-maker. A frozen 33M-parameter sentence-transformer embeds text descriptions of the state, each candidate action, and recent actions, and a compact multilayer perceptron scores each candidate. This is the per-action value network of He et al. (2016): because the action is an input, one network handles an action set that changes at every step. We characterise the NetSecGame environment and identify three properties that shape agent design and evaluation. Knowledge only grows; the actions available depend on what has been discovered, and their number grows quadratically with the host count; and in the deterministic, defender-free configuration used for prototyping, the task reduces to a Markov decision process. There, a greedy policy either always succeeds or always fails, so we report steps-to-goal rather than success rate alone. Training uses a Double-DQN-style target whose next action is chosen and cached when a transition is collected, not re-chosen at replay. In single runs on the simplest scenario, the agent learns the full attack chain (network scanning, service discovery, exploitation, data discovery, exfiltration) from a sparse terminal reward, and with linear exploration decay reaches the optimal five-step trajectory. Whether the semantic front end helps agents generalise to new network layouts, its intended benefit, remains untested, as does its contribution over a non-semantic baseline; we outline experiments for both.