Intelligent Web Security Systems: A Literature Survey on AI-Driven Vulnerability Detection and Automated Penetration Testing

Web Applications are an integral part of today's technological age, carrying out various applications in areas like healthcare, financial institutions, education, e-commerce, and government. Meanwhile, web application development has created a growing number and variety of cyber threats, with some of the most common being SQL injection (SQLi), XSS, CSRF, authentication bypasses, and more. The traditional cybersecurity mechanisms (e.g., vulnerability scanner tools, penetration testing) have proven to be inadequate, however, with respect to multiple factors that are inherent in the existing web environment, such as false positives, non-adaptability, lack of scalability, or a high level of human interaction. In this context, the recent developments in Artificial Intelligence (AI), machine learning (ML), Large Language Models (LLMs), and intelligent penetration testing (IPT) brought several novel solutions to the challenge of web application security and enabled intelligent and self-adaptive security systems. This paper, therefore, reviews the literature on the most recent developments in the area of intelligent web security, such as AI vulnerability detection, intelligent penetration testing, intelligent scanners and multi-agent systems, and the application of LLMs in cybersecurity. This paper aims to provide an introduction to the AI-powered web security system and discuss the need for adaptive and scalable and intelligent cybersecurity approaches to enhance the effectiveness of vulnerability detection.