Evaluation Workflows for Large Language Models (LLMs) that Integrate Domain Expertise for Complex Knowledge Tasks

The growing use of Large Language Models (LLMs) in specialized fields such as healthcare, nutrition, and education has raised critical concerns regarding the accuracy, reliability, and contextual appropriateness of LLM outputs.However, evaluating LLMs is challenging due to the complexity of the information and the need for human input, which is often costly and resource-intensive.My dissertation addresses the challenges in integrating domain expertise into the evaluation of LLM outputs for complex knowledge tasks to build more efficient evaluation workflows.The main objectives of this research are: 1) to investigate when and at what stage domain expertise should be integrated into LLM evaluation 2) to explore the role of domain experts compared to other evaluators such as lay users and LLMs themselves in the evaluation process, and 3) to design evaluation frameworks and tools that guide both the optimal integration of domain experts and leverage the complementary strengths of other evaluation groups.The expected impact of this research includes advancing the design of LLM evaluation tools and workflows that assist developers in identifying where expertise is needed to effectively develop and deploy LLMs in real-world applications. CCS Concepts