A hybrid intelligence approach to qualitative data analysis combining manual and AI coding with mixed methods analyses of reliability and validity

Abstract We conducted a methodological study of hybrid coding using qualitative data analysis (QDA) software. Manual coding of a corpus of 164 qualitative interviews (752,388 words) on meaningful media experiences was compared with AI coding in two prompting conditions: AI prompting with manual coding instructions vs. collaborative prompt engineering in dialogue with AI. Reliability and validity were assessed with a mixed methods design involving quantitative and qualitative analyses. Quantitative inter-coder agreement was low. However, qualitative analyses revealed that most relevant segments were correctly identified by AI with collaborative prompt engineering, but substantial overcoding occurred, concerning both the number of coded segments and segment length. AI coding also identified relevant segments that were overlooked during manual coding. Based on these findings, we cannot recommend AI coding as a replacement for manual coding. Rather, we propose a hybrid intelligence approach to QDA which involves prompt engineering in collaboration with AI, quantitative and qualitative validation analyses. Such hybrid coding can meaningfully complement manual analysis and augment its reliability and validity.