Measuring What Matters: Evaluating Ensemble LLMs with Label Refinement in Inductive Coding

Inductive coding traditionally relies on laborintensive human efforts, who are prone to inconsistencies and individual biases.Although large language models (LLMs) offer promising automation capabilities, their standalone use often results in inconsistent outputs, limiting their reliability.In this work, we propose a framework that combines ensemble methods with code refinement methodology to address these challenges.Our approach integrates multiple smaller LLMs, fine-tuned via Low-Rank Adaptation (LoRA), and employs a moderatorbased mechanism to simulate human consensus.To address the limitations of metrics like ROUGE and BERTScore, we introduce a composite evaluation metric that combines code conciseness and contextual similarity.The validity of this metric is confirmed through correlation analysis with human expert ratings.Results demonstrate that smaller ensemble models with refined outputs consistently outperform other ensembles, individual models, and even large-scale LLMs like GPT-4.Our evidence suggests that smaller ensemble models significantly outperform larger standalone language models, pointing out the risk of relying solely on a single large model for qualitative analysis.