AI in Usability Testing

This Master’s thesis investigates the potential of artificial intelligence (AI) in usability testing, focusing on how large language models can support or complement human evaluators. A custom API was developed using GPT-4o to analyze screenshots of online unit converter websites and identify usability problems based on Nielsen’s heuristics. The results were compared with evaluations from university students who reviewed the same websites as part of a usability course. Findings show that AI can efficiently detect common issues such as visibility problems, weak error prevention, and insufficient system feedback. Human evaluators, however, contributed valuable contextual and experiential insights, particularly in workflow efficiency and design clarity. The study highlights the strengths and limitations of both approaches and proposes a hybrid usability testing model that combines AI automation with human interpretation. This research contributes to the growing field of AI-driven usability evaluation by presenting empirical evidence, a working GPT-4o–based API, and recommendations for hybrid frameworks that balance scalability, efficiency, and human-centered design.