Efficient and effective web application security testing via reinforcement learning

Web application security is a persistent problem, increasing with the number of internet users and applications. Automated tools and techniques are commonly used to scan, test, or fuzz for bugs and vulnerabilities. However, the use of random search, manual heuristics, and the failure to use feedback makes existing work ineffective. Furthermore, the large number of test cases commonly needed to find comparatively small number of vulnerabilities makes it inefficient. With a problem-first approach this thesis examines the ability of Reinforcement Learning (RL) to perform tasks in web application security testing, including the generation and mutation of test cases in black- and grey-box settings. RL is a machine learning approach where an agent take actions in an environment to maximise long-term rewards overtime. Thus, the agent learns to complete a task. RL has achieved better-than-human performance in complex tasks ranging from game playing to protein design. Yet, the application of RL in web security is limited: relying on off-the-shelf RL models, simple heuristics, and domain knowledge to guide learning. We design bespoke RL agents with rich latent representations. We develop a hierarchical RL approach to cross-site scripting payload generation to constrain the challenge for agents. We then design a single agent that generates SQL injection payloads using latent representations in a changing action space. Furthermore, we evaluate the ability of RL to mutate HTTP test cases to find bugs in REST APIs. Finally, we present the first study to investigate where to perform mutations in JavaScript test cases, designing deep contextual bandit we place inside existing tools improves the efficiency of test case validity, coverage, and coverage-per-test case. Our RL agents are effective, finding 5 Cross Site Scripting (XSS) CVEs, 6 SQL injection CVEs, and 42 bugs in REST APIs. The agents are also efficient, reducing the number of attempts.