Firefly: Shedding Light on Verification Gaps using LLM-Powered Mutation Testing

Design verification remains one of the most challenging and resource-intensive stages in hardware development, with verification quality often dependent on the experience and judgment of engineers. Despite decades of research and tool development, functional bugs continue to escape to silicon, leading to costly respins and delayed product cycles. This paper introduces Firefly, a mutation testing framework that leverages large language models (LLMs) to inject context-aware faults into hardware designs. By evaluating whether a given verification infrastructure can detect these injected faults, Firefly reveals hidden weaknesses in the test suite and coverage model.We evaluate Firefly on two processors with contrasting architectures and verification strategies: Maveric Core 2, a five-stage RV64I pipeline, and Bitty, a 16-bit multicycle educational core. Both designs achieve high conventional coverage metrics—95% and 90% combined line and toggle coverage, respectively. However, Firefly reveals that a significant fraction of injected faults remain undetected: 28% of bugs went undetected in Maveric Core 2, and 67% in Bitty. These results highlight the limitations of traditional coverage-based assessment and underscore the need for deeper infrastructure qualification. Firefly addresses this gap through LLM-driven mutation testing, providing a lightweight, non-intrusive, and open-source solution that integrates seamlessly into modern verification workflows.