Conformance suites that pass but do not pin: mutation testing of agent receipt verifiers

A conformance suite for a receipt or acceptance-record verifier is usually judged by one number: how many vectors an implementation agrees with. That number says nothing about how many of the specification's rules the vectors actually test. I apply mutation testing to two independently written checkers for two public agent-evidence formats. Each mutant removes or loosens one behaviour of the checker; a behaviour is pinned if at least one vector disagrees after the change. Both checkers agree with every vector they are given. For the first, a hand-written battery of 28 rule-level mutations leaves 14 survivors against a 50-vector corpus and 10 against a 54-vector corpus. For the second, an automatic battery of 1,421 operator-level mutants leaves 306 survivors against 57 vectors, 48 of them in the profile's own verification steps. The unpinned behaviour clusters in the same places in both: key validity beyond the cases a vector happens to exhibit, number and key canonicalization, and fields whose values the vectors never vary. I list the survivors that a vector could pin, describe a counting rule that stops a failing checker from inflating its score, and argue that a suite should publish a mutation score next to its agreement count.Code and per-mutant results: https://github.com/actaseal/x402-acceptance-check-cold (Apache-2.0).

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-24
DOI
https://doi.org/10.5281/zenodo.23610509
Primary Topic
Software Testing and Debugging Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Conformance suites that pass but do not pin: mutation testing of agent receipt verifiers

Xavier Goshi
Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
preprint

Conformance suites that pass but do not pin: mutation testing of agent receipt verifiers

Xavier Goshi
preprint en

Abstract

A conformance suite for a receipt or acceptance-record verifier is usually judged by one number: how many vectors an implementation agrees with. That number says nothing about how many of the specification's rules the vectors actually test. I apply mutation testing to two independently written checkers for two public agent-evidence formats. Each mutant removes or loosens one behaviour of the checker; a behaviour is pinned if at least one vector disagrees after the change. Both checkers agree with every vector they are given. For the first, a hand-written battery of 28 rule-level mutations leaves 14 survivors against a 50-vector corpus and 10 against a 54-vector corpus. For the second, an automatic battery of 1,421 operator-level mutants leaves 306 survivors against 57 vectors, 48 of them in the profile's own verification steps. The unpinned behaviour clusters in the same places in both: key validity beyond the cases a vector happens to exhibit, number and key canonicalization, and fields whose values the vectors never vary. I list the survivors that a vector could pin, describe a counting rule that stops a failing checker from inflating its score, and argue that a suite should publish a mutation score next to its agreement count.Code and per-mutant results: https://github.com/actaseal/x402-acceptance-check-cold (Apache-2.0).

Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon