AI chatbots and cybersecurity: What do they tell the public?

This study evaluates the performance of five large language model chatbots – ChatGPT, Copilot, Gemini, Claude, and Cohere – on topics related to cybersecurity, healthcare, and the environment. The chatbots were evaluated by asking five specific questions, and their responses were analyzed to determine how well they aligned with the cybersecurity principles outlined in the Paris Call. To assess the semantic similarity of their responses, 384-dimensional sentence embeddings from Hugging Face (HuggingFace.com) were used to calculate cosine distances, offering a quantitative measure of their alignment with the Paris Call principles. Repeated measures of analysis of variance (ANOVA) revealed no significant differences in how frequently the chatbots applied the nine Paris Call principles to individual questions, with similar application rates across the chatbots. However, a separate ANOVA across all five questions identified significant differences (p = 0.011) in the average use of these principles, suggesting that the chatbots likely rely on different datasets and did not consistently apply the principles across all questions. The study also found errors of omission, where certain key principles were left out of some responses. For example, several chatbots failed to mention critical elements, e.g. protecting the integrity of supply chains or ensuring accountability in technology use, highlighting gaps in their cybersecurity coverage. As a result, users may need to query multiple chatbots to gain comprehensive insights on these topics.