Adaptive Composite Accuracy Scoring for Domainspecific LLM Evaluation

With the increasing demand for Large Language Models (LLM) deployment across different industries, the need to evaluate LLMs with benchmarking datasets prior to deployment becomes apparent. Accuracy scores are computed individually after testing the LLMs with multiple benchmarking datasets. The popular publicly available general benchmarking datasets, such as the Massive Multitask Language Understanding (MMLU) dataset, cover multitude of domains, which are less suitable for domain-specific LLM use cases. This paper combines the goodness of objective-based evaluation (using benchmarking datasets) and subjective-based evaluation (using LLM-as-a-judge) and devises an adaptive weightage-based method to compute a domain-specific Weighted Composite Accuracy Score (WCAS) extracted from the existing benchmarking datasets without the need to rerun the entire benchmarking datasets. With a more refined domain-specific Weighted Composite Accuracy Score (WCAS), the result shows that the methodology has effectively eliminated 47 MMLU datasets with zero relevancy to the specific domain and achieved accuracy refinement of up to 12.4% and time savings of up to 87.2% compared with running 57 MMLU benchmarking datasets. This approach accelerates the decision-making process in selecting the most suitable LLM for domain-specific deployment.