Pre-registration: paired decision eval ("right for the wrong reasons") – EuConform, stage 1

Pre-registration (stage 1) of the paired decision eval in EuConform, an open-source evidence toolkit for AI compliance. The instrument asks whether a language model is right for the right reason: every synthetic base case is paired with decisive, irrelevant, placebo and paraphrase variants, scored in both option orders, and the analysis compares accuracy (Acc) with grounded accuracy (GA). This deposit fixes, before any model has been run: hypotheses H0–H4 with refutation criteria, metrics, the six open 0.5–2B models with Hugging Face revisions, rule families, frozen prompt wording, name lists, seeds, the pre-registered model pairs, and the generator manifests with the SHA-256 of the dev and pilot prompt sets. Stages 2 and 3 (label tokens per model, N and k from the power simulation, test split hashes) are recorded later in the repository, with deviations dated. Claim boundary: "for six open 0.5–2B models on a synthetic rule-following task with eight rule families". The task is clinically motivated but synthetic; it is not medical advice and makes no clinical claim. Source: github.com/Hiepler/EuConform, commit 650cf91f78fef68533e4c4c3d08f8b3b794be82c on the main branch (merged with merge commit ac8c1f2c3666cfca8039ed53409ff88f4b04a431). Verify the archive with shasum -a 256 -c SHA256SUMS. SHA-256 of the archive euconform-paired-eval-prereg-stage1.zip: 93d6f94854a6d93bac331a7335e2be578ae5cb955464ba77e626d188e355bd85. The prompt-set hashes in manifests/ equal those in the pre-registration and can be reproduced with euconform paired-eval generate.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-11
DOI
https://doi.org/10.5281/zenodo.24134198
Primary Topic
Explainable Artificial Intelligence (XAI)
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Pre-registration: paired decision eval ("right for the wrong reasons") – EuConform, stage 1

Benedikt Hiepler
Zenodo (CERN European Organization for Nuclear Research)
Explainable Artificial Intelligence (XAI)
article

Pre-registration: paired decision eval ("right for the wrong reasons") – EuConform, stage 1

Benedikt Hiepler
article en

Abstract

Pre-registration (stage 1) of the paired decision eval in EuConform, an open-source evidence toolkit for AI compliance. The instrument asks whether a language model is right for the right reason: every synthetic base case is paired with decisive, irrelevant, placebo and paraphrase variants, scored in both option orders, and the analysis compares accuracy (Acc) with grounded accuracy (GA). This deposit fixes, before any model has been run: hypotheses H0–H4 with refutation criteria, metrics, the six open 0.5–2B models with Hugging Face revisions, rule families, frozen prompt wording, name lists, seeds, the pre-registered model pairs, and the generator manifests with the SHA-256 of the dev and pilot prompt sets. Stages 2 and 3 (label tokens per model, N and k from the power simulation, test split hashes) are recorded later in the repository, with deviations dated. Claim boundary: "for six open 0.5–2B models on a synthetic rule-following task with eight rule families". The task is clinically motivated but synthetic; it is not medical advice and makes no clinical claim. Source: github.com/Hiepler/EuConform, commit 650cf91f78fef68533e4c4c3d08f8b3b794be82c on the main branch (merged with merge commit ac8c1f2c3666cfca8039ed53409ff88f4b04a431). Verify the archive with shasum -a 256 -c SHA256SUMS. SHA-256 of the archive euconform-paired-eval-prereg-stage1.zip: 93d6f94854a6d93bac331a7335e2be578ae5cb955464ba77e626d188e355bd85. The prompt-set hashes in manifests/ equal those in the pre-registration and can be reproduced with euconform paired-eval generate.

Zenodo (CERN European Organization for Nuclear Research)
Explainable Artificial Intelligence (XAI)
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon