Evaluating the Effectiveness of LLM in Measuring Students' Foreign Language Conversational Abilities in the Information Technology Teacher Interview

This study evaluates the effectiveness of large language models (LLMs) in assessing foreign language conversational abilities during IT teacher interviews. Comparing ChatGPT with human evaluators across syntactic accuracy, vocabulary richness, content completeness, and answer relevance, results show high consistency in overall scores (ICC = 0.952, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$p$</tex> <0.001). However, significant differences emerged in vocabulary richness (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$p&lt;0.001$</tex>) and answer relevance (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$p&lt;0.001$</tex>), highlighting AI's strengths in lexical analysis but limitations in contextual understanding. These findings underscore LLMs' potential as supplementary tools in high-stakes assessments, with implications for refining AI-driven evaluation frameworks.