AutoTestAI: Generating Software Test Cases from Screen Recordings using Optical Character Recognition and Vision-Language Models

Browser automation tools like Selenium and Playwright are reliable for running tests. However, tests still need to be written manually and linked to Document Object Model (DOM) locators, which can break if the page layout changes. We introduce AutoTestAI, a system that takes a screen recording of someone performing a workflow and generates test cases that can be turned into Playwright scripts. The system does not need access to the application's source code or API. Users upload screen recordings through a React dashboard to a FastAPI backend. OpenCV samples the video, Tesseract extracts on-screen text, and keyframes are chosen when the text changes. Each keyframe is described by LLaVA 7B, running locally via Ollama, using the recognised text in the prompt. Deterministic rules then create draft test cases for the tester to review, and a template-based generator produces the Python Playwright scripts. In a pilot, only 3 of 20 extracted frames were sent to the vision model, and 3 test cases were generated and stored. Across 5 runs on a laptop with an Intel Core i5-1335U processor and no GPU, the pipeline averaged 31.12 s per run, with 89% of the time spent in the vision model. A generated login script passed all 3 runs, with an average test time of 9.86 s. Text recognition was the weakest part, with a character error rate of 32.04% and a word error rate of 49.65% on 5 hand-transcribed frames. Measuring action-recognition accuracy and test-case correctness will require an annotated dataset, which is planned for future work.