Pre-registration: paired decision eval ("right for the wrong reasons") – EuConform, stage 1
Pre-registration (stage 1) of the paired decision eval in EuConform, an open-source evidence toolkit for AI compliance. The instrument asks whether a language model is right for the right reason: every synthetic base case is paired with decisive, irrelevant, placebo and paraphrase variants, scored in both option orders, and the analysis compares accuracy (Acc) with grounded accuracy (GA). This deposit fixes, before any model has been run: hypotheses H0–H4 with refutation criteria, metrics, the six open 0.5–2B models with Hugging Face revisions, rule families, frozen prompt wording, name lists, seeds, the pre-registered model pairs, and the generator manifests with the SHA-256 of the dev and pilot prompt sets. Stages 2 and 3 (label tokens per model, N and k from the power simulation, test split hashes) are recorded later in the repository, with deviations dated. Claim boundary: "for six open 0.5–2B models on a synthetic rule-following task with eight rule families". The task is clinically motivated but synthetic; it is not medical advice and makes no clinical claim. Source: github.com/Hiepler/EuConform, commit 650cf91f78fef68533e4c4c3d08f8b3b794be82c on the main branch (merged with merge commit ac8c1f2c3666cfca8039ed53409ff88f4b04a431). Verify the archive with shasum -a 256 -c SHA256SUMS. SHA-256 of the archive euconform-paired-eval-prereg-stage1.zip: 93d6f94854a6d93bac331a7335e2be578ae5cb955464ba77e626d188e355bd85. The prompt-set hashes in manifests/ equal those in the pre-registration and can be reproduced with euconform paired-eval generate.
Authors
- Benedikt Hiepler (ORCID: https://orcid.org/0009-0005-2068-0415)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-11
- DOI
- https://doi.org/10.5281/zenodo.24134198
- Primary Topic
- Explainable Artificial Intelligence (XAI)
- Type
- article
- Field-Weighted Citation Impact
- 0.00