RAG Effectiveness Across Model Size and Architecture: A Cross-Family Evaluation on a Domain-Specific Croatian Accounting Benchmark

This paper presents a systematic empirical evaluation of Retrieval-Augmented Generation (RAG) across 22 open-weight language models spanning five families—Qwen3.5, Llama 3.x, Gemma 3, Phi-4, and DeepSeek R1 distilled—on a domain-specific Croatian accounting benchmark derived from the publicly available mariozupan/rrif dataset. All models were deployed locally via Ollama on a single GPU server, reflecting privacy-preserving enterprise conditions. Two retrieval corpora were evaluated: a structured English Q&Aderived corpus (Corpus A, 522 accounts) and a Croatian corpus parsed from the RRiF chart of accounts (Corpus B, 2462 entries), the latter indexed under three embedding models, yielding five conditions and 116,600 queries in total. Four findings are reported. First, on Corpus A, RAG raises lookup Exact Match from near-zero to 94.9–96.2% with 18 of the 22 models converging into that band, with equivalence testing confirming practical equivalence for 87% of within-band pairs at a two-point margin; the four exceptions lie at 1B–3B parameters, but the boundary tracks instruction-following quality rather than scale. Because Corpus A shares its source with the evaluation questions, these results are an upper bound rather than an estimate of field performance. Second, embedding model choice dominates language model choice on the Croatian corpus, where a multilingual encoder raises average Exact Match by approximately 43 percentage points. Third, reasoningdistilled models show no consistent advantage on this task. Fourth, decomposing responses into mutually exclusive outcome categories shows that retrieval reduces the invention of account codes from 33.6% to below 0.3% under every condition tested, while residual error consists almost entirely of wrong-but-existing accounts whose source shifts from retrieval to selection as retrieval improves—a distinction that a binary hallucination measure conflates. Non-generative baselines outperform every model on Corpus A and ten of the 22 on Corpus B, indicating that a language model is warranted only conditionally for this task.