Learning to Detect Hardcoded Secrets

Numerous tools have been developed to uncover secrets and credentials in source code. However, they often rely on predefined rules that yield a high rate of false positives. We conduct an in-depth investigation into the performance of different AI models in secret detection. In non-generative models, we found that the choice of feature set greatly influences the performance in secret detection. These models demonstrated the best performance using the context feature, i.e., code surrounding the secret value. Particularly, CodeBERT outperforms other models with an MCC and F1-scores of 88% and 89%, respectively.When we applied the models to an unseen dataset of secrets, the top models were RF and CodeBERT, achieving an MCC score of around 65%. In generative models, we observed moderate performance (a maximum MCC of 63\%) and a high number of false positives, most of which occurred in the test path files. Our manual investigation of incorrect predictions revealed that LLMs tend to be cautious, flagging potential secrets to promote secure coding practices.