IoTBenchSL: A Streamlined Framework for the Efficient Production of Standardized IoT Benchmarks with Automated Pipeline Validation

Security vulnerabilities in IoT firmware pose significant risks, and black-box protocol fuzzing offers a scalable and cost-effective solution for identifying these vulnerabilities. However, current fuzzing evaluation schemes are often beset by challenges such as limited benchmark dataset sizes, low benchmark production efficiency, insufficient automation, and poor adaptability, which hinder cross-method comparisons. To address these challenges, we propose IoTBenchSL, a modular, pipeline-driven framework for constructing standardized, reproducible, and integrable vulnerability detection benchmark datasets. IoTBenchSL introduces a benchmark production workflow alongside a CI/CD-based automated validation pipeline, equipping each benchmark with a one-click emulation environment, automated validation mechanisms, and exploit verification scripts. This approach enhances the efficiency and accuracy of benchmark generation. Additionally, IoTBenchSL seamlessly integrates with existing fuzzing frameworks, enabling large-scale fuzzing evaluations and vulnerability discovery without modifying fuzzers. In our experiments, IoTBenchSL was used to generate 100 firmware benchmarks and 100 vulnerability benchmarks. We identified 32 previously unknown vulnerabilities through these benchmarks, 21 of which received CVE or CNVD identifiers. These results demonstrate IoTBenchSL’s capacity to generate large-scale IoT fuzzing benchmarks, improving evaluation efficiency and reducing costs for comparative studies, thereby advancing reproducible fuzzing research.