Methodological implications of using machine learning to estimate the impact of AI on the workforce

Examining the potential effects of artificial intelligence on jobs has been a research topic for many years, carrying significant implications for social and industrial policies. Frey and Osborne's seminal study, which estimated that AI could potentially displace 47 % of jobs, has inspired numerous subsequent studies that have reused many elements of the original research. However, the methodological approach and application of machine learning in their study has largely escaped critical examination. Given the study's significant influence in both academic circles and public discourse, this article aims to offer a methodological critique of Frey and Osborne's work and their use of machine learning to assess how these factors may have shaped their findings and conclusions. The analysis finds that their study lacks the necessary methodological robustness to produce reliable results and that the use of machine learning to estimate the impact of AI on the workforce would not be recommended. Additionally, this paper briefly explores the similarities with recent studies on the impact of generative AI on the workforce, highlighting comparable methodological issues. As a result, this paper proposes a future research agenda to help researchers, policymakers, and businesses gain a better understanding of how AI technologies may impact the workforce. • Critical analysis of methodological approaches in AI job impact studies reveals significant robustness issues • Questions reliability of machine learning applications for workforce displacement estimates • Identifies similar methodological issues in recent studies on generative AI’s workforce impact. • Proposes a new research agenda for understanding of AI’s effects on jobs for researchers, policymakers, and businesses.