Memorization in LLM-Based Program Repair

Automated Program Repair (APR) is a powerful technique for mitigating the impact of software bugs in software development. The recent remarkable success of Large Language Models (LLMs) has set new state-of-the-art performance in APR. However, the extensive use of large training corpora raises concerns about whether these impressive capabilities genuinely generalize to unseen tasks or primarily rely on memorizing vast amounts of pretraining data. To address this issue, this paper introduces a memorization-inducing prompting strategy, MemInducer, to investigate the extent of memorization in LLMs for APR. Specifically, MemInducer is designed to prompt LLMs to recall responses from their training corpus. Subsequently, we assess memorization by measuring the similarity between the responses generated by the LLM and the corresponding ground truth. Experimental results reveal that memorization is indeed present in existing APR benchmarks, with over 78% of corrected bugs producing results that are entirely identical to the ground truth.