Cevra Reasoning Decisions 4B System Card: How Does Reference-Based Assessment Compare with Jev?

Cevra Reasoning Decisions is a reference-based assessment system that scores a candidate answer and rationale against a supplied reference on a 0–1 scale. This system card summarizes saved evaluations of the 4B model in merged BF16 and Ollama Q4 representations, alongside a trained 0.6B system and comparator models, using mean absolute error (MAE) against available target scores. On 113 held-out synthetic HLE-derived examples, the trained 4B BF16 and Q4 representations achieved MAE 0.0145 and 0.0091, respectively, each with 113/113 coverage; the generated-score Qwen3.5-4B baseline achieved MAE 0.0066 on 113/113 examples under a different scoring protocol. The synthetic targets were strongly concentrated at 1: 110/113 targets were 1, two were 0.75, and one was 0.25; consequently, a constant score of 1 achieved MAE 0.0111 on 113/113 examples. On 51 selected original HLE examples, all targets were 1, yielding MAE 0.0000 for the constant-score-1 baseline; the 4B BF16 and Q4 representations achieved MAE 0.0817 and 0.0717 on 51/51 examples. Because training data were derived from HLE Diamond, the selected original-HLE evaluation is not an independent generalization test. These results characterize agreement with the available target scores under differing scoring procedures and coverage, rather than general reasoning ability or reliable discrimination of incorrect answers.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24345142
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Cevra Reasoning Decisions 4B System Card: How Does Reference-Based Assessment Compare with Jev?

Evgenii Evstafev
1 links
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Cevra Reasoning Decisions 4B System Card: How Does Reference-Based Assessment Compare with Jev?

Evgenii Evstafev
preprint en

Abstract

Cevra Reasoning Decisions is a reference-based assessment system that scores a candidate answer and rationale against a supplied reference on a 0–1 scale. This system card summarizes saved evaluations of the 4B model in merged BF16 and Ollama Q4 representations, alongside a trained 0.6B system and comparator models, using mean absolute error (MAE) against available target scores. On 113 held-out synthetic HLE-derived examples, the trained 4B BF16 and Q4 representations achieved MAE 0.0145 and 0.0091, respectively, each with 113/113 coverage; the generated-score Qwen3.5-4B baseline achieved MAE 0.0066 on 113/113 examples under a different scoring protocol. The synthetic targets were strongly concentrated at 1: 110/113 targets were 1, two were 0.75, and one was 0.25; consequently, a constant score of 1 achieved MAE 0.0111 on 113/113 examples. On 51 selected original HLE examples, all targets were 1, yielding MAE 0.0000 for the constant-score-1 baseline; the 4B BF16 and Q4 representations achieved MAE 0.0817 and 0.0717 on 51/51 examples. Because training data were derived from HLE Diamond, the selected original-HLE evaluation is not an independent generalization test. These results characterize agreement with the available target scores under differing scoring procedures and coverage, rather than general reasoning ability or reliable discrimination of incorrect answers.

Zenodo (CERN European Organization for Nuclear Research)
University of Cambridge (GB)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon