LEDGER: Deterministic Checks and Auditable Evaluation for Business AI Systems

Abstract Public benchmarks rank models on public tasks, but those scores do not directly predict performance on a company's own work. Companies need a practical way to retest their AI systems after every change. We present LEDGER, an evaluation platform that turns a company's own data, questions, and rules into a repeatable test. Deterministic checks score what they can, and a separate LLM judge and a deterministic evidence check handle the rest, with human review of sampled answers. Every score traces to evidence, and cost per correct answer makes the tradeoff between accuracy and cost explicit. Because each failure points to a scenario, expected value, and rule, LEDGER also guides improvement: coding agents repair the system while sampled human review validates what counts as correct. Interval used it to build its business intelligence assistant around DeepSeek V4 Flash (DS4), a privately hosted open-weight model. In Interval's harness, DS4 passes 96 out of 97 held-out scenarios, while five proprietary models and the open-weight GLM-5.3 pass 94 to 97 given the same tools and definitions. Even under their generally cheaper schema-only prompt, they cost an estimated 3 to 54 times as much per correct answer as bare self-hosted DS4. LEDGER makes this tradeoff measurable, showing how a company can achieve frontier-level accuracy while choosing its own balance of cost and control.