LLM Trust Score: Scoring the Reliability of Language-Model Answers with Cross-Model Judges and Self-Consistency

Large language models (LLMs) often give fluent answers that are wrong, and users have no cheap way to tell which answers to trust. This paper presents LLM Trust Score, a black-box layer that scores one answer from 0 to 100 without access to token probabilities. A writer model (gpt-4o-mini) answers each question three times at temperature 0.9. A Gemini model checks whether the three answers reach the same conclusion. Three judges from three other developers (Qwen, Mistral and Llama) rate each answer as correct or hallucinated with the help of web-search snippets, and the best-scoring answer receives a score and a High, Medium or Low label. A separate tie-breaker model (NVIDIA Nemotron) checks only the winning answer and removes 10 points when it disagrees with the judge majority. We evaluate on 200 fresh questions (66 TruthfulQA, 66 HaluEval and 68 TriviaQA) graded by Claude Haiku 5.5, a model that plays no part in the scoring. The writer was correct on 61.0% of questions. The trust score separated correct from incorrect answers with an AUROC of 0.674 (95% bootstrap interval 0.604–0.740) and a mean gap of 18.1 points (11.1–25.5). The labels were ordered as intended: 71.1% of High, 55.0% of Medium and 26.3% of Low answers were correct. The score is not a calibrated probability. Of the answers that scored 100, 28.0% were wrong, and the Brier score (0.272) was worse than always predicting the base rate (0.238). Discrimination was weakest on TruthfulQA (AUROC 0.568, interval 0.450–0.682). A simple proxy that uses only the judges’ hallucination flags ranked answers at least as well as the full score (AUROC 0.721). Reading the tie-breaker’s penalty back from the scores, it disagreed with the judges on 17 answers (8.5%), of which 35.3% were correct against 63.4% for the rest, and it added only 0.013 to the AUROC (interval −0.003 to +0.033). We give the reasons for every design choice, including why a Gemini agreement checker was preferred to a natural-language-inference (NLI) model, and we state which components were not evaluated on their own.