Evaluating the Performance of Large Language Models for Design Validation

With the emergence of Large Language Models (LLMs), there has been a growing interest in harnessing their potential applications beyond traditional natural language processing tasks. One such application is hardware design validation. This paper presents a comprehensive evaluation of LLMs in design validation tasks. In design validation, it is essential to analyze the designs and crafting appropriate testbenches for them. We evaluate the ability to recognize hardware descriptions as well as the ability to generate testbenches for those designs. We present evaluation methodology and benchmarks to evaluate these tasks. Experiments were conducted with four prominent LLMs and designs ranging from small arithmetic block up to a small MIPS CPU. The results demonstrate promising performance for a limited complexity threshold.