PrincipalBench v2: A Validity-First Benchmark for LLM Orchestrators in Multi-Agent Pipelines

Multi-agent systems place a principal model in charge of decomposing work, supervising worker agents, and synthesising their outputs, yet orchestrators are still chosen with benchmarks that measure single-model task performance. PrincipalBench was introduced to measure orchestration directly. This report describes its second version. We first audit the v1 release against its own code and data and find that it cannot support model comparisons: 433 of its 500 tasks are templated placeholders without real questions or gold answers, every evaluation episode contained an injected failure so a policy that flags everything attains perfect detection, only three distinct failures were ever injected, the context-coherence metric was computed from worker outputs and is therefore identical for every model, and two-thirds of injected failures came from a different domain than the task. A model-free "flag everything" policy scores 0.52 PrincipalScore under the v1 harness. PrincipalBench v2 replaces the protocol with a two-phase sealed design: planning is scored from the question alone; supervision is scored on gold-aligned worker reports in which half of the episodes are clean controls and failures are derived from the hop's own gold output. Detection is chance-corrected (Youden's J), the final step is withheld so the orchestrator must synthesise the answer, and coherence is measured on the orchestrator's own ledger. The v2 core split contains 67 hand-authored tasks (289 subtasks, 335 episodes over five seeds). Reference policies now behave as a benchmark should: degenerate strategies score 0.00 on detection and at most 0.50 PrincipalScore, while only a full oracle reaches 1.00. We withdraw the model rankings reported with v1 and release v2 as the basis for future, reproducible orchestrator evaluations. Code and data are available at https://github.com/KalarisLabs/Principal-Bench.