Deep Learning-Based Automatic Usability Testing for Detecting Frustration and Confusion in User Interaction

Real-time and scalable usability testing methods are the need of the hour as the digital systems evolve to be more adaptive and user focused. Conventional usability assessment methods, including interview and post-task query, are time-consuming, observer-biased and cannot measure emotional response in real-time. This study introduces a deep learning model for automatic usability evaluation using user frustration and confusion detection with the help of affective computing in realistic scenario. We present a CNN-LSTM, hybrid model to capture spatio-temporal emotional features from facial images of video sequences using the DAiSEE dataset with frame-wise user affect annotations in e-learning domain. The model obtained good classification results (average F1-score of 85.3% and the ROC-AUC of 0.91 and 0.89 for frustration and confusion). To connect technical progress to potential user needs, the framework was implemented in a prototype interface and assessed in a 20 user UX study. The results showed that the system’s usability score was strong, the trust level was moderate, and the ethical issues were manageable, especially in terms of emotional data privacy. Our study suggests that emotion detection based on deep learning has the potential to improve the usability evaluation process; it is scalable to multiple users, can be used in real-time and from an ethical perspective, sensitive to users’ feelings, making a step towards more adaptive and a human-aware digital systems.