Can (A)I help you? Comparing human and GenAI analysis of HCI qualitative research results

Generative AI (GenAI) is experiencing rapid growth, particularly in its application as a tool for qualitative text analysis—a key element of Human-Computer Interaction (HCI) research. This study examines the potential of GenAI, specifically ChatGPT, to assist in the analysis of qualitative research data. Four qualitative HCI studies, previously conducted and analyzed by our research group, were selected for this investigation. ChatGPT was employed to perform AI-assisted analyses on the raw data from these studies, and the AI-generated insights were then compared with the human-led analyses already completed. The results reveal significant alignment between the human and AI-assisted analyses, indicating that GenAI can serve as an effective support tool in qualitative research. However, while GenAI offers considerable advantages in enhancing research efficiency, human oversight remains crucial to ensuring accurate interpretation and contextual alignment. This study also provides practical recommendations for researchers interested in incorporating GenAI into their qualitative analysis processes.