ASTRAL: A Tool for the Automated Safety Testing of Large Language Models

In this paper, we present ASTRAL, a tool that automates the generation and execution of test inputs (i.e., prompts) to evaluate the safety of Large Language Models (LLMs). ASTRAL consists of three microservice modules. The first is a test generator, which employs a novel black-box coverage criterion to create balanced and diverse unsafe test inputs across a wide range of safety categories and linguistic characteristics (e.g., different writing styles and persuasion techniques). Additionally, the test generator incorporates an LLM-based approach that leverages Retrieval-Augmented Generation (RAG), few-shot prompting strategies, and web browsing to produce up-to-date test inputs. The second module is the test executor, which runs the generated test inputs on the LLM under test. Finally, the test evaluator acts an oracle to assess the execution outputs to identify unsafe responses, enabling a fully automated LLM testing process.