Leveraging Pre-Trained Large Language Models (LLMs) for On-Premises Comprehensive Automated Test Case Generation: An Empirical Study

The rapidly evolving field of Artificial Intelligence (AI)-assisted software testing has predominantly focused on automated test code generation, with limited research exploring the realm of automated test case generation from user stories requirements. This paper presents a comprehensive empirical study on harnessing pre-trained Large Language Models (LLMs) for generating concrete test cases from natural language requirements given in user stories. We investigate the efficacy of various prompting and alignment techniques, including prompt chaining, few-shot instructions, and agency-based approaches, to facilitate secure on-premises deployment. By integrating our learnings with an on-premises model setup, wherein we deploy a RoPE scaled 4-bit quantized LLaMA 3 70B Instruct model, optionally augmented with LoRA adapters trained on QA datasets, we demonstrate that this approach yields more accurate and consistent test cases despite video random-access memory (VRAM) constraints, thereby maintaining the security benefits of an on-premises deployment.