Conformance suites that pass but do not pin: mutation testing of agent receipt verifiers
A conformance suite for a receipt or acceptance-record verifier is usually judged by one number: how many vectors an implementation agrees with. That number says nothing about how many of the specification's rules the vectors actually test. I apply mutation testing to two independently written checkers for two public agent-evidence formats. Each mutant removes or loosens one behaviour of the checker; a behaviour is pinned if at least one vector disagrees after the change. Both checkers agree with every vector they are given. For the first, a hand-written battery of 28 rule-level mutations leaves 14 survivors against a 50-vector corpus and 10 against a 54-vector corpus. For the second, an automatic battery of 1,421 operator-level mutants leaves 306 survivors against 57 vectors, 48 of them in the profile's own verification steps. The unpinned behaviour clusters in the same places in both: key validity beyond the cases a vector happens to exhibit, number and key canonicalization, and fields whose values the vectors never vary. I list the survivors that a vector could pin, describe a counting rule that stops a failing checker from inflating its score, and argue that a suite should publish a mutation score next to its agreement count.Code and per-mutant results: https://github.com/actaseal/x402-acceptance-check-cold (Apache-2.0).
Authors
- Xavier Goshi (ORCID: https://orcid.org/0009-0006-0583-6968)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-10-24
- DOI
- https://doi.org/10.5281/zenodo.23610509
- Primary Topic
- Software Testing and Debugging Techniques
- Type
- preprint