Copilot in service: Exploring the potential of the large language model-based chatbots for fostering evaluation culture in preventing and countering violent extremism

<ns3:p>Background The rapid advancement in artificial intelligence (AI) technology has introduced the large language model (LLM)-based assistants or chatbots. To fully unlock the potential of this technology for the preventing and countering violent extremism (P/CVE) field, more research is needed. This paper examines the feasibility of using chatbots as recommender systems to respond to practitioners’ needs in evaluation, increase their knowledge about the key evaluation aspects, and provide practical guidance and professional support for the evaluation process. At the same time, the paper provides an overview of the limitations that such solution entails. Methods To explore the performance of the LLM-based chatbots we chose a publicly available AI assistant called Copilot as an example. We conducted a qualitative analysis of its responses to 50 pre-designed prompts of various types. The study was driven by the analysis questions established to explore accuracy and reliability, relevance and integrity, as well as readability and comprehensiveness of the responses. We derived the key aspects of evidence-based evaluation along with practitioners’ needs from the results of the H2020 INDEED project. Results Our findings indicate that Copilot demonstrated significant proficiency in addressing issues related to evidence-based evaluation in P/CVE. Most generated responses were factually accurate, relevant, and structurally sound, i.e. sufficient to kick-start and deepen internal evidence-based practise. At the same time, biases and data security issues inherent in LLM-based chatbots should be carefully explored by practitioners. Conclusions This study underscored both the potential and limitations of LLM-based chatbots in fostering evaluation culture in P/CVE. While Copilot can effectively generate accessible, informative and encouraging recommendations, it still requires a professional oversight to manage and coordinate the evaluation process, as well as address more field-specific needs. The future research should focus on more rigorous and user-centred assessment of such systems for P/CVE use based on multidisciplinary efforts.</ns3:p>