Algorithmic Inversion: A Learnable Algorithm Representation for Code Generation

The prevalent fine-tuning paradigm for large language models (LLMs) has demonstrated strong performance on various code generation tasks. However, these models still fall short when confronted with algorithmic programming problems, where precise algorithmic reasoning is required. Humans typically adopt diverse algorithmic techniques to tackle complex programming problems, enabling general analysis and accurate implementation. Building on this observation, we propose a method that learns compact, LLM-friendly representations of algorithmic knowledge, termed Algorithmic Inversion (AI), which aims to aid LLMs in understanding programming problems. Specifically, we apply a lightweight fine-tuning process on codeoriented models to automatically learn algorithm embeddings. When concatenated with the inputs, the algorithm embeddings act as instructive signals, guiding LLMs in generating correct code solutions by providing contextual algorithmic hints. We apply our approach to models of three different parameter sizes and evaluate them on three algorithmic programming benchmarks. Our extensive experiments show that applying AI to small (1.5B parameters) models results in absolute improvements of up to 1.8 on Pass@1, while large models (1 5 B parameters) achieve improvements of up to 1.4, compared to Prompt-Tuning. Additionally, our method outperforms traditional full fine-tuning approaches by a significant margin across all tested benchmarks. Furthermore, our analysis of the generated code reveals that AI effectively enhances the model's problem-solving process by providing clear algorithmic guidance. Codes and datasets are available<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup><sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>https://github.com/joeysbase/Algorithmic-Inversion .