On Prompt Learning for FQN Inference: Sensitivity and Usefulness Analysis

The success of prompt learning when adapted to the fully-qualified type name (FQN) inference has been demonstrated in the literature. However, the understanding of its success is limited in model outputs and model structures. In this paper, we conduct a thorough study on the behaviors of prompt learning in FQN inference from the perspectives of sensitivity and usefulness. Rather than simply masking some knowledge, we first perform sensitivity analysis on five aspects to reveal how much FQN knowledge to include, how much to mask, and where to mask, and then yield an efficient configuration strategy. We further conduct a usefulness analysis in three aspects to demonstrate the superiority of the proposed configuration strategy. This suggests that the strong performance of our model is attributable to the homogeneity among large code pre-training, FQN prompt learning, and type inference as a fill-in-blank task. Finally, we summarize a practical guideline on best practices and pitfalls to avoid when applying prompt learning to FQN inference and other software engineering tasks.