AutoPT: How Far Are We From the Fully Automated Web Penetration Testing?

Penetration testing is essential for ensuring Web security by identifying and mitigating vulnerabilities in advance, and the rapid progress of large language models (LLMs) shows great potential to revolutionize this process through intelligent, automated agents. In this work, we establish a comprehensive end-to-end penetration testing benchmark using a real-world penetration testing environment to explore the capabilities of LLM-based agents in this domain. Our results reveal that the agents are familiar to procedures of penetration testing tasks, but they still face limitations in generating accurate commands and executing complete processes. Accordingly, we summarize the current challenges, including the difficulty of maintaining the entire message history and the tendency for the agent to become stuck. Based on the above insights, we propose a Penetration testing State Machine (PSM) that utilizes the Finite State Machine (FSM) methodology to address these limitations. Then, we introduce AutoPT, an automated penetration testing agent based on the principle of PSM driven by LLMs, which utilizes the inherent inference ability of LLM and the constraint framework of state machines. Our evaluation results show that AutoPT outperforms the the ReAct-based baseline and improves the task completion rate from 22% to 41% on the benchmark target. Compared with the baseline and manual work, AutoPT also reduces time and economic costs further. In general, our AutoPT has facilitated the development of automated penetration testing and bring new findings and insights for both academia and industry.