Can generative AI detect and fix real-world cryptographic misuses?

We evaluate ChatGPT’s ability to detect and fix cryptographic API misuses. First, we show that GPT-4o, when guided by engineered prompts, achieves F1 scores above 0.90 on two established benchmarks. Next, we assess the model on Java samples collected from GitHub repositories and Android apps. On the GitHub samples, it attains an F1 score of 0.84 and a Matthews Correlation Coefficient (MCC) of 0.81; on the Android samples, it achieves the F1 of 0.85 but a slightly lower MCC of 0.76. We found that factors beyond the misuse type such as file, method, and variable names; file size; code comments/context; the complexity of API usage (e.g., basic vs. interprocedural, field-sensitive, path-sensitive); and the number of misuses per file, affect ChatGPT’s detection performance. For comparison, we apply a leading crypto misuse detector to the Android apps and observe lower performance (F1 = 0.81, MCC = 0.68). We conclude that ChatGPT is a reliable tool for uncovering cryptographic misuses. Moreover, when it flags a misuse, it can suggest effective fixes. We reported identified cryptographic API misuses in GitHub repositories, each with GPT-generated fix suggestions. Maintainers responded to 23 misuses, with 21 confirming the issues and agreeing with the proposed fixes. However, several declined to implement changes due to legacy compatibility, hardware limitations, or because the code serves as a tutorial or sample. In two cases, developers did not consider the identified misuses to be security issues, as the code was intended for non-sensitive data. • ChatGPT has a promising performance in cryptographic misuse detection • ChatGPT, guided by engineered prompts, outperforms the leading static crypto misuse detector • ChatGPT’s recommendations for crypto misuses are mostly correct and secure.