Does an Integrity-Gated Governance Observability Index Measure Reconstructability? A Simulation-Based Criterion-Validity Study of Agent Telemetry under Out-of-Specification Faults
Governance of agentic AI systems depends on execution evidence: an operator can judge whether an agent stayed within its authority only if its telemetry allows what it did to be reconstructed. The Governance Observability Index (GOI) proposes to measure this as a gated geometric mean of coverage, fidelity, integrity and timeliness, with an integrity gate intended to stop untrustworthy evidence being masked by completeness elsewhere. This simulation study tests one implementation of the index against criteria computed from complete ground truth, of which the estimator sees only a 40-record ledger sample. A seeded testbed simulates a scripted payments agent, a policy gateway, an authoritative ledger and an HMAC-chained telemetry stream. The estimator uses only the stream, a gateway control total and a 40-record ledger sample. Twelve fault families—four within the original specification, four outside it and four benign infrastructure perturbations—are evaluated in 2,520 single-family runs and 1,000 mixed runs. GOI correlates with reconstruction accuracy (Spearman ρ = 0.85, 95% CI 0.83–0.87) and separates degraded from sound reconstructions (AUC 0.93). It is weaker as a warning of missed time-sensitive violations (AUC 0.87), and 63% of runs scoring at least 0.90 had degraded reconstructions. The integrity gate, as implemented, never activated in the mixed runs. A 5% forgery by a key holder left the index at 0.99 while 98% of sequence-level violations went undetected in time. When deletion targets the same records as forgery, it shifts part of the defect from fidelity to coverage; a sensitivity experiment with independent deletion leaves the headline results unchanged. In this testbed GOI works as a screening measure, not a certificate of trustworthy evidence, and its usefulness depends on evidence channels independent of the telemetry.
Authors
- Ravi Shankar NRK
Publication Details
- Journal
- Research Square
- Published
- 2026-11-19
- DOI
- https://doi.org/10.21203/rs.3.rs-11552307/v1
- Primary Topic
- Software System Performance and Reliability
- Type
- preprint