ICARuS: Intercode-CTF Auto-Randomization System

Large Language Models (LLMs) have seen rapid advancements since 2020 and have shown impressive capabilities in the domain of software development. This has raised both curiosity and concerns surrounding the cybersecurity capabilities of LLMs. Recent work has sought to quantify these capabilities by adapting publicly available capture-the-flag (CTF) challenges into a benchmark. While CTF challenges are an attractive choice for benchmarking LLM cybersecurity capabilities due to their self-contained nature across a wide-range of cyber-related skills, there is a risk of test-set contamination where the LLMs have been trained on publicly-available solutions. To measure the extent that LLMs can adapt to variations in these types of problems, we present ICARuS, a randomization framework that generates randomized instances of tasks defined in a previously published benchmark (InterCode-CTF). In addition, we estimate the complexity of each Intercode-CTF task and show model performance degradation is inversely correlated with the number of steps required to solve the problem.