Cevra Reasoning Decisions 4B System Card: How Does Reference-Based Assessment Compare with Jev?
Cevra Reasoning Decisions is a reference-based assessment system that scores a candidate answer and rationale against a supplied reference on a 0–1 scale. This system card summarizes saved evaluations of the 4B model in merged BF16 and Ollama Q4 representations, alongside a trained 0.6B system and comparator models, using mean absolute error (MAE) against available target scores. On 113 held-out synthetic HLE-derived examples, the trained 4B BF16 and Q4 representations achieved MAE 0.0145 and 0.0091, respectively, each with 113/113 coverage; the generated-score Qwen3.5-4B baseline achieved MAE 0.0066 on 113/113 examples under a different scoring protocol. The synthetic targets were strongly concentrated at 1: 110/113 targets were 1, two were 0.75, and one was 0.25; consequently, a constant score of 1 achieved MAE 0.0111 on 113/113 examples. On 51 selected original HLE examples, all targets were 1, yielding MAE 0.0000 for the constant-score-1 baseline; the 4B BF16 and Q4 representations achieved MAE 0.0817 and 0.0717 on 51/51 examples. Because training data were derived from HLE Diamond, the selected original-HLE evaluation is not an independent generalization test. These results characterize agreement with the available target scores under differing scoring procedures and coverage, rather than general reasoning ability or reliable discrimination of incorrect answers.
Authors
- Evgenii Evstafev
Institutions
- University of Cambridge (GB)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.5281/zenodo.24345142
- Primary Topic
- Topic Modeling
- Type
- preprint