Intent-driven Web UI Tests Repair with LLM

As web applications are frequently updated, changes may introduce to web elements in new versions, causing test cases to fail. Consequently, automatic web test repair techniques are proposed to reduce the cost of regression testing. Most existing methods focus on finding the correct candidate elements or related attributes to fix the broken test case. However, when test case failures are caused by test flow changes or propagated breakages, those methods that focus solely on matching the failing element in the new version cannot work well. Through empirical analysis, we found that the test intent and the reasons that caused the test failure are useful in test case reparation. This paper proposes a novel intent-driven web test repair approach named LetTe, which first parses the test intent and failure reasons of failed web UI tests, and then guides the Large Language Model (LLM) to fix them via prompt design and fine-tuning. LetTe’s repair logic is to simulate that of a human expert. According to the test intent and reason for failure, “think about” the possible repair plan and then generate repair candidates through the corresponding chain-of-thought. We evaluate LetTe on 7 web applications collected from open-source websites as well as publicly available datasets. The experimental results show that our approach has a 75% correct repair rate, which is higher than all baseline methods.