The Dark Side of Generative Artificial Intelligence: Abuse and Vulnerabilities

Abstract. Generative Artificial Intelligence (GenAI) is a field of study that encompasses the development of large-scale models trained on billions of parameters. These models are utilized to generate content across various media, primarily relying on publicly available data as input. Such models have already been deployed in several industries, including banking, healthcare, education, and insurance. Gartner (2024) posits that by 2027, over 50% of enterprises will employ industry-specific GenAI models, a significant increase from the 1% that utilized such models in 2023. Additionally, Gartner (2024) projects that investments in GenAI and cybersecurity will grow by over 80% in 2025 compared to 2024. Guaranteeing security and robustness of GenAI systems is essential to utilize it effectively, particularly in the areas of significant societal importance, such as critical infrastructure. The potential risks associated with the deployment of GenAI systems have been identified since the initial stages of their development. Since these systems evolve, the need for a more profound comprehension of their functionality in real-world contexts and identification of potential misuse increases. Most of the research in this field focuses on hypothetical scenarios and mapping potential risks. This provides the foundation for analyzing threats and vulnerabilities in the AI domain. Nevertheless, one of the few studies examining actual attacks demonstrates that many cases of GenAI system misuse do not occur due to sophisticated technical attacks but rather through the exploitation of readily available functionalities that require minimal technical expertise (Marchal et al., 2024).