Anxiety in an artificial intelligence (AI)-based English conversation test: An idiodynamic approach to explore potential construct-irrelevant variance

The present study explored anxiety and its fluctuation during an artificial intelligence (AI)-based English conversation test as a test validation study. Applying an idiodynamic approach, 13 Japanese university students learning English as a foreign language (EFL) took the test and participated in pre- and post-test interviews, which were analyzed by conducting reflexive thematic analysis. Individual differences in the pattern of anxiety experience fluctuations were observed, while factors affecting their anxiety experiences were also explored. These were classified into either authentic factors observed in both testing and non-testing contexts (e.g., perceived English proficiency, perceived communication skills, a two-way relationship between cognition and anxiety, and features of conversation) or factors unique to the test (evaluative situation and AI’s unnatural behavior). From the perspective of test validation, this study emphasized the importance of paying attention to construct-irrelevant variance, such as anxiety unnecessarily arising from test-specific factors.