Detecting Smart Contract Vulnerabilities using Transformers and LLMs

This study investigates the detection of vulnerabilities in smart contracts using various transformer models and Large Language Model (LLM) systems. We evaluated BERT, CodeBERT, DistilBERT, and the Gemini model, employing techniques such as aggregation of chunks to enhance performance. The results indicate that simple transformers applied to source code generally perform worse than when applied to byte-code. However, the use of aggregation techniques on the source code significantly improved the model performance. We also evaluate the use of meta-classifiers for multimodal data, by stacking multiple transformers working on source code and byte-code. The Random Forest meta-classifier achieved the highest performance but exhibited significant overfitting. The Gemini model demonstrates limited performance, highlighting the necessity of proper training for LLM systems.