Pay Attention! Human-Centric Improvements of LLM-based Interfaces for Assisting Software Test Case Development

Implementing automation testing is difficult and as a consequence there is a growing desire for semi-automated software testing systems with humans in the loop. Leveraging the growth of LLMs, recent research has demonstrated LLMs’ potential to improve performance on test generation, reporting, and bug triaging. However, relatively little work has explored the interactivity issues that emerge in semi-automated LLM-assisted software test case development. To fill this gap, we present two user studies (N1 = 16, N2 = 24) that investigate productivity, creativity, and user attention in three semi-automated LLM-assisted interaction strategies: (1) pre-emptive prompting; (2) buffered response; and (3) guided input. We find that pre-emptively prompting the user significantly enhances branch coverage and task creativity by more than 30% while reducing user’s off-task idle time by up to 48.7%. We conclude by suggesting concrete research directions applying mixed-initiative principles for LLM-based interactive systems for semi-automated software testing.