Automating content analysis of scientific abstracts using ChatGPT: A methodological protocol and use case

This paper presents a protocol for using ChatGPT to perform content analysis. The protocol involves converting a codebook, outlining categories, descriptions, coding rules, and possible values, into a structured prompt that guides ChatGPT's analysis. The protocol was validated through analysis of 980 research articles to identify research approaches and data collection methods. ChatGPT achieved high performance in identifying data collection methods, but faced challenges with poorly defined or underrepresented categories, particularly in mixed methods research. Overall, while it scored well for quantitative (0.96) and qualitative (0.82) studies, it struggled with mixed methods (0.60), highlighting the need for clear methodological definitions.•The protocol enhances coding efficiency and demonstrates the feasibility of using AI for content analysis, potentially streamlining the coding process in research.•Challenges arose in categories that were not clearly defined (big data), underrepresented (ethnography), or hierarchically related (Interview & Discourse/Textual analysis).•Interrater metrics indicated a substantial level of agreement, reinforcing the potential of ChatGPT in content analysis while emphasizing the importance of clear methodological definitions.