When Are Decision Models Worth It? An Empirical Comparison with Small Task-Specific Classifiers

Preprint, not peer reviewed. Decision models such as OpenAI Decisions and Base Labs' Elephant-1 and Mammoth-1 return a probability for each option of a closed question and need no task-specific training. This independent study asks when they are worth using instead of small classifiers trained on the task. We compare them with TF-IDF, SetFit, mmBERT and Qwen3 classifiers (0.6B, 1.7B, 8B parameters) on Banking77 (English), MASSIVE it-IT (Italian) and a synthetic Italian customer-service task, varying the label budget (5 and 20 examples per class, full training set) and the category descriptions, and testing a lexical stress set and a phenomenon-level robustness set. With few labelled examples the decision models are far ahead, even when given label names only; with the full training sets and the protocol recipes the gap is 2.4 points on Banking77, 5.5 on Italian MASSIVE and not resolved on the synthetic task, and on Banking77 a different fine-tuning recipe leaves no resolved difference, so in this regime the outcome depends on the training recipe. The effect of category descriptions varies with model and dataset, and Elephant-1 is much closer to its own base model, Qwen3.5-9B, than to an older generalist. The study uses a frozen protocol, paired bootstrap intervals and three seeds; code, per-example predictions and synthetic data are released. Limitations: no real customer messages or human annotations, synthetic data written by language models, possible contamination of public benchmarks, endpoints evaluated in October 2026. The author set the research question and the requirements of the study; an AI coding agent (Claude Code) proposed the detailed protocol, wrote the code, ran the experiments and drafted the manuscript under the author's direction. See the "AI assistance and authorship" section of the paper.