Shift or Shimmer? The Promises of Generative AI for Automated Content Analysis in Marketing

This paper explores the contributions and limitations of Generative AI (GenAI), including Chat GPT, in automated content analysis (ACA) in marketing. It compares the performance of several Chat GPT-driven ACA protocols with manual and automated ad hoc dictionary protocols, highlighting methodological and practical implications. The experiment uses a corpus of 992 responses from a consumer survey. It examines the quality of the different protocols for coding verbatim, but also for the design of thematic grids and ad hoc dictionaries . Performance is evaluated on validity and efficiency criteria. On the various key tasks of ACA, Chat GPT performs significantly worse than classical methods based on corpus analysis and researcher participation. The results show that GenAI has reduced validity, with frequent omissions and errors in the coding of specific data. GenAI can intervene in certain stages of ACA and offer time savings, but it requires rigorous and informed human supervision to ensure the validity of the results, highlighting the importance of a hybrid human-machine approach. The article provides a first systematic evaluation of GenAI applied to ACA in marketing, and proposes methodological recommendations, highlighting the fundamental differences between the algorithmic approach and the researchers’ approach based on real data.