Large Language Models, Fine Tuning, Code Generation

Automated software synthesis, text-to-code (code-generation), can be an important aid to software developers. Developments in this area have existed practically since the advent of computing. The new artificial intelligence methods, deep learning (DL), and one type of them, large language models (LLMs) emerged in the last decade, represent a new opportunity for creating this type of application. We survey the transformer technology that underlies the operation of language models. Present paper briefly reviews the topic of software synthesis and code generation, briefly mentioning the main trends. The document concludes with an empirical study demonstrating the described results, where the T5 language model is fine-tuned to generate SQL queries. A fine-tuning set of a few thousand examples train T5 to produce quality queries. The economic principle of diminishing returns (Gossen I) applies, and it is not worth using too large a fine-tuning set. In fact, it is the attractiveness of fine-tuning.