An Evaluation Framework for Post‐Trained Bookkeeping Language Models

ABSTRACT Supervised fine‐tuning (SFT) enables large language models (LLMs) to acquire domain‐specific knowledge, yet the resulting trade‐off between specialization and general capability preservation remains poorly understood. This paper investigates this trade‐off systematically by fine‐tuning three open‐source LLMs on double‐entry bookkeeping posting schemes and evaluating them against a comprehensive benchmark suite. The study contrasts full parameter fine‐tuning with parameter‐efficient QLoRA across dense and Mixture‐of‐Experts architectures, under real‐world GPU memory constraints. An industry‐standard evaluation framework ensures reproducible and transparent benchmarking alongside a custom domain‐specific task. Results suggest that the selected fine‐tuning configuration can have a more pronounced effect on outcomes than model size alone, with implications for practitioners selecting strategies for domain‐specific LLM deployment.