Attributing ChatGPT-Generated Source Codes

AI assistants such as ChatGPT have remarkable human-like capabilities, producing natural language and programming language utterances. Despite that, ChatGPT could facilitate academic misconduct by easily generating codes and text as solutions for assignments. More alarmingly, ChatGPT can be used to write polymorphic malware. Moreover, ChatGPT-generated codes are shown to be less secure. While the detection of text generated by ChatGPT has been addressed, ChatGPT code authorship attribution is largely unexplored. In this article, we examine attributing ChatGPT codes using off-the-shelf code authorship attribution techniques. We demonstrate that the answer to the question is negative, necessitating a new approach, which we also deliver by scrutinizing the outcomes of the off-the-shelf attribution technique. We found that grouping ChatGPT codes using the inference step of a pretrained model on non-ChatGPT codes can be used as an accurate attribution model. Compared with the 8.3%–29.2% accuracy of the naive approach, our approach delivers 81.3%–91.7% while costing a small trade-off in the accuracy of detecting target (non-ChatGPT) authors. Moreover, the straightforward authorship attribution model trained for the binary classification (ChatGPT versus <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Human</i>) achieved a classification accuracy of 87% with 6 K code samples. Our comprehensive analysis sheds light on the limitations of the styles generated by ChatGPT, making detecting codes generated by ChatGPT feasible.