Model Editing for LLMs4Code: How Far are We?

The Code directory contains all experimental code. To reproduce the experiments, please refer to the README file within the Code directory. The Results directory contains all experimental results. The EditConala and EditCodeSearchNet folders contain results for the CNLE and CSNE datasets, respectively. In the AGRACE folder: run_A_GRACE.json contains the results for A-GRACE. run_w_o_CL.json contains the results without CL. run_w_o_mean.json contains the results without the mean. The agrace_encoder_data folder includes trained A-GRACE encoders. **_train_eparam.pt is the weight of A-GRACE's encoder. **_train_last_token_eparam.pt is the weight when not using CL. The prompt templates used to construct datasets are as follows: Prompt template to evaluate the semantic matches between the code and the description: I'm filtering a python code dataset, and I want to get high-quality data from it. I will give you an Intent that represents the intention for generating code and a Snippet that is a fragment of the desired code. Please rate the degree of match between the Intent and the Snippet, with scores ranging from 0 to 4. The scoring rules are as follows: 0: Very Irrelevant. 1: Irrelevant. 2: Neutral. 3: Relevant. 4: Very Relevant. # EXAMPLES # # 1 Intent: sort a nested list l by two elements '1' and '0' Snippet: sorted(l, key=lambda x: (-int(x[1]), x[0])) Score:4 # 2 Intent: make curvilinear plots in matplotlib Snippet: plt.show() Score:1 # RESPONSE # Response a score between 0 to 4 with no more explanations. # INSTANCE # Intent: {} Snippet: {} Score: Prompt template to generate rewritten intents: You need to rephrase an intent statement (i.e., 'Intent') that generates Python code (i.e., 'Snippet') to create a new sentence (i.e., 'Rephrased Intent'), ensuring that this sentence is different from the original intent statement but conveys the same semantics. The intention statement and the new statement are both aimed at generating code snippets. # EXAMPLE # Snippet: sorted(l, key=lambda x: (-int(x[1]), x[0])) Intent: sort a nested list l by two elements '1' and '0' Rephrased Intent: rearrange the nested list 'l' considering two elements, '1' and '0' # INSTANCE # Snippet: {} Intent: {} Rephrased Intent: