Test-Agent: A Multimodal App Automation Testing Framework Based on the Large Language Model

This paper introduces a multimodal agent-based app automation testing framework, named Test-Agent, built on the Large Language Model (LLM), designed to address the growing challenges in mobile application automation testing. As mobile applications become more prevalent and emerging systems like Harmony OS Next and mini-programs emerge, traditional automated testing methods, which depend on manually crafting test cases and scripts, are no longer sufficient for cross-platform compatibility and complex interaction logic. The Test-Agent framework employs artificial intelligence technologies to analyze application interface screenshots and user natural language instructions. Combined with deep learning models, it automatically generates and executes test actions on mobile devices. This innovative approach eliminates the need for pre-written test scripts or backend system access, relying solely on screenshots and UI structure information. It achieves cross-platform and cross-application universality, significantly reducing the workload of test case writing, enhancing test execution efficiency, and strengthening cross-platform adaptability. Test-Agent offers an innovative and efficient solution for automated testing of mobile applications.