Leveraging Large Language Model Outputs for Enhanced Domain Specific Decision-Making

Data annotation is crucial for training machine learning models through supervised learning, with the quality, consistency, and quantity of annotations significantly impacting model performance [18]. However, achieving high-quality annotations at scale typically requires substantial investments. Large Language Models (LLMs) offer a promising solution for large-scale annotation, but they have limitations such as inconsistency [1], task-specific unsuitability [1], and inherent training biases [2]. Rather than attempting to eliminate these biases, we propose leveraging them to create a more informed annotation process. We introduce an approach that employs multiple LLMs to gather diverse perspectives, addressing the limitations of using a single model. By combining these perspectives using our proposed strategies— Exclusive Union, Inclusive Intersection, and Expert LLM approaches—we create comprehensive annotations that capitalize on each model’s strengths. Additionally, we validated our method through a small-scale, domain-specific project, demonstrating its practical feasibility and benefits with minimal human intervention. This approach can be scaled to larger projects, offering a cost-effective and efficient solution for high-quality data annotation that leverages the unique capabilities of multiple state-of-the-art LLMs.