Factors determining the efficacy of AI-generated word problems for content-specific math language courses in higher education

The use of artificial intelligence (AI) tools to generate content-specific instructional materials has attracted the interest of Language for Specific Purposes (LSP) educators in higher education, as language courses in this setting typically do not utilize a textbook, requiring the instructor to create independent materials. However, instructors are often not content experts. Collaboration between LSP instructors and content experts in the form of co-teaching is one way in which materials can be generated and benefit instructors and students alike. At the same time, creating instructional materials can be a time-consuming task and can detract from other areas of collaboration. The use of AI tools to generate mathematical content could help instructors save time, enable real-world connections and offer a variety of materials to students. This study examines the generation of word problems with the help of AI tools for a content-based mathematics language course for first-semester bachelor students pursuing a Science, Technology, Engineering or Mathematics (STEM) degree. As they form part of the final exam, a new set of word problems needs to be generated each year. While recent studies (cf. Lu et al., 2022) found that AI methods were effective in generating math word problems that were diverse, relevant, and useable, there have been no studies examining the applicability of AI-generated word problems in terms of their efficacy in an LSP setting. The action research study used screencasting to capture the math content tutors’ content analysis of AI output on math word problems and was followed by a semi-structured group interview. The results showed that word problems generated by AI were generally useful based on factors such as prompting techniques. However, limitations were observed in the areas of accuracy and consistency. Based on initial results, the report suggests first implications for use in LSP instruction and describes measures that need to be taken into account in further studies.