Smaller Large Language Models for Text-To-Sql: Performance Analysis and Optimal Performance

Text-to-SQL aims to transform natural language queries into executable SQL, a task in which Large Language Models (LLMs) have demonstrated significant success. However, their extremely high computational demands often make practical implementation challenging. This study investigates the performance of smaller models, ranging from 0.5B to 3.8 B parameters, after fine-tuning with Parameter-Efficient Fine-Tuning (PEFT) techniques, namely LoRA and QLoRA. Experimental results indicate that such small models can achieve execution accuracy as high as <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$7 6. 8 3 \%$</tex>, comparable to much larger models such as Mistral, Gemma-2, and Qwen2.5 Coder 7B, while requiring significantly fewer computational resources. These findings highlight the feasibility of smaller models for real-world Text-to-SQL applications and challenge the prevailing assumption that high accuracy and efficiency are attainable only through large-scale models.