Coft: Making Large Language Models Better Zero-Shot Learners for Code Generation

The Chain-of-Thought (CoT) prompting mechanism has effectively enhanced the performance of large language models (LLMs) across a variety of natural language processing (NLP) tasks, including complex zero-shot learning scenarios. Recent studies suggest that this effectiveness arises from CoT's capacity to direct LLMs' attention toward task-relevant keywords. However, traditional CoT methods yield only marginal improvements in the realm of code generation, particularly for models with fewer than 10 billion parameters. We posit that this limitation stems from the substantial disparity between the logical structure and representational form of code compared to natural language. Considering the training and deployment costs, enhancing the performance of small LLMs through advanced prompting and instruction-tuning is essential. In this paper, we introduce COFT (Chain of Functional Triggers), a novel prompting strategy specifically designed for code generation tasks. The design of COFT is based on the following important observation: An optimal CoT tailored for code generation should clearly indicate the core functionality of each critical step, while employing standard identifiers prevalent within the coding domain. Extensive experiments conducted on representative small LLMs (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$&lt;10 \mathrm{B}$</tex>) benchmarks demonstrate that our COFT substantially outperforms vanilla CoT methods. In challenging zeroshot scenarios and the Pass@1 metric, COFT can improve the performance of fundation LLMs by up to 35.3 %. These empirical findings support our hypothesis that an appropriate design for CoT alongside instruction tuning can fully activate even smallersized LLMs, making them better zero-shot learners for code generation. The source code of COFT and the constructed instruction-tuning dataset will be released.