Evaluation of LLM-based chatbots for OSINT-based Cyber Threat Awareness

Knowledge sharing about emerging threats is crucial in the rapidly advancing field of cybersecurity and forms the foundation of Cyber Threat Intelligence (CTI). In this context, Large Language Models (LLMs) are becoming increasingly significant in the field of cybersecurity. This study surveys the performance of ChatGPT, GPT4all, Dolly, Stanford Alpaca, Alpaca-LoRA, Falcon, and Vicuna LLM-based chatbots in binary classification and Named Entity Recognition (NER) tasks using Open-source intelligence (OSINT) to detect and extract structured data about cybersecurity threats. We utilize data collected in previous research from Twitter to assess the competitiveness of these chatbots when compared to specialized state-of-the-art models trained for those tasks. In binary classification experiments, the commercial chatbot GPT-4 achieved an acceptable F 1 score of 0.94, and the open-source GPT4all achieved an F 1 score of 0.90. However, when applied for cybersecurity entity recognition, all evaluated LLM-based chatbots have limitations and are less effective. This study demonstrates the capability of LLM-based chatbots for OSINT processing and shows that they require further improvement in NER to effectively replace specially trained models. Our results highlight their strengths and limitations compared to specialized models. This provides insights for researchers to enhance LLM-based chatbots to reduce the effort required to integrate machine learning in OSINT-based CTI tools. • Presenting a survey on chatbot models and their application to cybersecurity. • Inherent flexibility of chatbots for specific requirements of CTI applications. • Highlighting the strength, limitations of chatbots compared to specialized models.