What counts as evidence in AI & ED: Towards Science-for-Policy 3.0

Since the 1990s, there have been heated debates about how evidence should be used to guide teaching practice and education policy, and how educational research can generate robust and trustworthy evidence. This paper reviews existing debates on evidence-based education and research on the impacts of AI in education and suggests a new conceptualisation of evidence aligned with an emerging learning-oriented model of science-for-policy, which we call S4P 3.0. Existing empirical evidence on AIED suggests some positive effects, but a closer look reveals methodological and conceptual problems and leads to the conclusion that existing evidence should not be used to guide policy or practice. AI is a new type of technology that interacts with human cognition, communication, and social knowledge infrastructures, and it requires rethinking what we mean by “learning outcomes” and policy and practice-relevant evidence. A common belief that AI-supported personalisation will “revolutionise” education is historically rooted in a methodological confusion that we call the Bloomian paradox in AIED, and based on a limited view on the social functions of education.