Does an Integrity-Gated Governance Observability Index Measure Reconstructability? A Simulation-Based Criterion-Validity Study of Agent Telemetry under Out-of-Specification Faults

Governance of agentic AI systems depends on execution evidence: an operator can judge whether an agent stayed within its authority only if its telemetry allows what it did to be reconstructed. The Governance Observability Index (GOI) proposes to measure this as a gated geometric mean of coverage, fidelity, integrity and timeliness, with an integrity gate intended to stop untrustworthy evidence being masked by completeness elsewhere. This simulation study tests one implementation of the index against criteria computed from complete ground truth, of which the estimator sees only a 40-record ledger sample. A seeded testbed simulates a scripted payments agent, a policy gateway, an authoritative ledger and an HMAC-chained telemetry stream. The estimator uses only the stream, a gateway control total and a 40-record ledger sample. Twelve fault families—four within the original specification, four outside it and four benign infrastructure perturbations—are evaluated in 2,520 single-family runs and 1,000 mixed runs. GOI correlates with reconstruction accuracy (Spearman ρ = 0.85, 95% CI 0.83–0.87) and separates degraded from sound reconstructions (AUC 0.93). It is weaker as a warning of missed time-sensitive violations (AUC 0.87), and 63% of runs scoring at least 0.90 had degraded reconstructions. The integrity gate, as implemented, never activated in the mixed runs. A 5% forgery by a key holder left the index at 0.99 while 98% of sequence-level violations went undetected in time. When deletion targets the same records as forgery, it shifts part of the defect from fidelity to coverage; a sensitivity experiment with independent deletion leaves the headline results unchanged. In this testbed GOI works as a screening measure, not a certificate of trustworthy evidence, and its usefulness depends on evidence channels independent of the telemetry.

Authors

Publication Details

Journal
Research Square
Published
2026-11-19
DOI
https://doi.org/10.21203/rs.3.rs-11552307/v1
Primary Topic
Software System Performance and Reliability
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Does an Integrity-Gated Governance Observability Index Measure Reconstructability? A Simulation-Based Criterion-Validity Study of Agent Telemetry under Out-of-Specification Faults

Ravi Shankar NRK
Research Square
Software System Performance and Reliability
preprint

Does an Integrity-Gated Governance Observability Index Measure Reconstructability? A Simulation-Based Criterion-Validity Study of Agent Telemetry under Out-of-Specification Faults

Ravi Shankar NRK
preprint en

Abstract

Governance of agentic AI systems depends on execution evidence: an operator can judge whether an agent stayed within its authority only if its telemetry allows what it did to be reconstructed. The Governance Observability Index (GOI) proposes to measure this as a gated geometric mean of coverage, fidelity, integrity and timeliness, with an integrity gate intended to stop untrustworthy evidence being masked by completeness elsewhere. This simulation study tests one implementation of the index against criteria computed from complete ground truth, of which the estimator sees only a 40-record ledger sample. A seeded testbed simulates a scripted payments agent, a policy gateway, an authoritative ledger and an HMAC-chained telemetry stream. The estimator uses only the stream, a gateway control total and a 40-record ledger sample. Twelve fault families—four within the original specification, four outside it and four benign infrastructure perturbations—are evaluated in 2,520 single-family runs and 1,000 mixed runs. GOI correlates with reconstruction accuracy (Spearman ρ = 0.85, 95% CI 0.83–0.87) and separates degraded from sound reconstructions (AUC 0.93). It is weaker as a warning of missed time-sensitive violations (AUC 0.87), and 63% of runs scoring at least 0.90 had degraded reconstructions. The integrity gate, as implemented, never activated in the mixed runs. A 5% forgery by a key holder left the index at 0.99 while 98% of sequence-level violations went undetected in time. When deletion targets the same records as forgery, it shifts part of the defect from fidelity to coverage; a sensitivity experiment with independent deletion leaves the headline results unchanged. In this testbed GOI works as a screening measure, not a certificate of trustworthy evidence, and its usefulness depends on evidence channels independent of the telemetry.

Research Square
Software System Performance and Reliability
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon