Selection, Not Scale: A Pre-Registered Ladder Study of a 0.6B Arabic MCQ Generator

Can a 0.6B model write Arabic reading-comprehension multiple-choice questions as well as akilx, a public 1.7B generator? We fixed the decision rules before any run, built the system as a ladder of five changes on Qwen3-0.6B, and spent $17.13 in cloud spend. On 460 ARCD passages the primary goal failed under the pre-registered protocol (final system minus akilx -3.5 points, 95% interval [-8.9, +2.0]); with options shuffled (paper 1's protocol) the difference is +0.0 [-5.7, +5.7]; the +5 primary criterion is not met under either. We stopped as the pre-registered failure rule required, and that decision stands. Two findings hold under both protocols. First, test-time selection does almost all of the work: best-of-5 with a verifier adds 23.5 points [+18.0, +28.9] (23.0 shuffled), while evidence-first prompting, constrained format and DPO are close to zero; the final 0.6B system is not significantly different from a tuned 1.7B model (-0.2 [-5.7, +5.0]; +1.7 shuffled). Second, our measurements misled us in five documented ways: an underpowered dev set reversed a real +12.8-point effect; wrong answer keys passed a human pass and, in one case, both judges; the good metric rewards questions that leak their answer; two leakage measures disagreed about DPO; and option order alone, a protocol detail lost between two studies of the same group, moved the headline comparison by 3.5 points and changed its outcome category.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-19
DOI
https://doi.org/10.5281/zenodo.24357371
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Selection, Not Scale: A Pre-Registered Ladder Study of a 0.6B Arabic MCQ Generator

Ahmed Othman
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Selection, Not Scale: A Pre-Registered Ladder Study of a 0.6B Arabic MCQ Generator

Ahmed Othman
preprint en

Abstract

Can a 0.6B model write Arabic reading-comprehension multiple-choice questions as well as akilx, a public 1.7B generator? We fixed the decision rules before any run, built the system as a ladder of five changes on Qwen3-0.6B, and spent $17.13 in cloud spend. On 460 ARCD passages the primary goal failed under the pre-registered protocol (final system minus akilx -3.5 points, 95% interval [-8.9, +2.0]); with options shuffled (paper 1's protocol) the difference is +0.0 [-5.7, +5.7]; the +5 primary criterion is not met under either. We stopped as the pre-registered failure rule required, and that decision stands. Two findings hold under both protocols. First, test-time selection does almost all of the work: best-of-5 with a verifier adds 23.5 points [+18.0, +28.9] (23.0 shuffled), while evidence-first prompting, constrained format and DPO are close to zero; the final 0.6B system is not significantly different from a tuned 1.7B model (-0.2 [-5.7, +5.0]; +1.7 shuffled). Second, our measurements misled us in five documented ways: an underpowered dev set reversed a real +12.8-point effect; wrong answer keys passed a human pass and, in one case, both judges; the good metric rewards questions that leak their answer; two leakage measures disagreed about DPO; and option order alone, a protocol detail lost between two studies of the same group, moved the headline comparison by 3.5 points and changed its outcome category.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon