Examining Trustworthiness of LLM-as-a-Judge Systems in a Clinical Trial Design Benchmark

Manual evaluation of Large Language Model (LLM) applications at scale presents significant resource challenges, making LLM-as-Judge (LaaJ) an attractive alternative. This study examines the reliability of LaaJ evaluation within CT-Bench, a benchmark for assessing LLMs’ capabilities in recommending clinical trial baseline features. LaaJ-alpha, our GPT-4o based prototype, semantically matches LLM-recommended features against reference features from clinical trials, accounting for semantic equivalence (e.g., ‘BMI’ and ‘Body Mass Index’). The system generates matched pairs and unmatched features from both sources to calculate precision, recall, and F1 scores. Laaj-alpha evaluates baseline feature recommendations across CTBench CT-Pub (100 trials) and CT-Repo (1,690 trials) for comparing results for GPT-4o and Llama-3-70B-Instruct under zero-shot and three-shot settings. Coherence checking revealed hallucinations in LaaJ-alpha’s evaluation, necessitating a post-processing correction step that yielded lower but more accurate performance metrics. Three different types of hallucination were observed. The hallucination rate provides a quantifiable coherence metric that can be systematically used to improve LaaJ reliability. Our findings underscore the challenges in developing reliable LLM evaluation methods in healthcare applications and demonstrate a potential framework for improving LaaJ systems.