Evaluating LLM-Based Communicative Agents for Verilog Design

Large language models have demonstrated their effectiveness in generating simple hardware descriptions for a variety of basic hardware design problems. One-shot generation, however, is unable to solve many problems reliably. By utilizing Communicative Agents and a compiler-in-the-loop feedback loop, the language model was shown to be more effective than in single-shot code generation. We present an experimentation framework for connecting all necessary infrastructure to track, extract, utilize, and evaluate large language models working in tandem with each other, and with tools like Icarus Verilog, to solve hardware design problems. Our results suggest that recently proposed approaches for communicative agent-based design are only marginally beneficial in a hardware design context.