Impact of Parameter-To-Data Ratio on LLM Fine-Tuning in Russian Text Classification Tasks

This paper addresses the optimization of fine-tuning large language models (LLMs) for Russian-language text classification under constrained computational resources. The proposed approach hinges on balancing the model size (i.e., number of parameters) against the volume of training data: a smaller model is fine-tuned on a larger dataset and compared against a larger model fine-tuned on a smaller dataset. The aim was to establish the impact of different ratios of model parameters and data for further training on the quality of text classification by large language models. We hypothesized that a weaker (i.e., smaller) model trained on more data could achieve classification performance comparable to or even surpassing that of a stronger (i.e., larger) model trained on less data. This hypothesis was motivated by the need to adapt LLMs to Russian-language tasks, where increased dataset size may compensate for reduced model capacity. The hypothesis was evaluated across three classification tasks: sentiment analysis of movie reviews, sentiment analysis of service reviews, and topic classification of news articles. The experiments were conducted on Russian-language datasets and employed the multilingual models XLM-RoBERTa-comet-small (107M parameters) for the weaker model and XLM-RoBERTa-base (278M parameters) for the stronger model. The smaller model was fine-tuned on proportionally larger datasets (scaled according to the parameter count difference) while the larger model used correspondingly smaller datasets. The weaker model consistently matched or exceeded the performance of the stronger model while requiring 2–3 times fewer computational resources (measured in FLOPs). The result highlights the practical value of this approach for energy-efficient fine-tuning in Russian-language settings.