Methods for Addressing the Societal Challenges of Malicious Narratives in Social Media Using AI/ML Analysis: a Case Study of Hate Speech

This paper highlights a methodology for analyzing and monitoring malicious communication in social media in order to mitigate the potential harm. There has been a deliberate "weaponization" of messaging through the use of social networks, including by politically oriented entities, both state-sponsored and privately run. The article identifies a use of AI/ML characterization of generalized "mal-info," a broad term which includes deliberate malicious narratives similar to hate speech, which adversely impacts society. A key point of the discussion is that this mal-info will dramatically increase in volume, and it will become essential for sharable quantifying tools to provide support for human expert intervention. Despite attempts to introduce moderation on major platforms like Facebook and X/Twitter, there are now established alternative social networks that offer completely unmoderated spaces. The paper presents an introduction to these platforms and a case study showing a qualitative and semi-quantitative analysis of representative examples of mal-info posts. The action examines several inflammatory terms using text analysis and, importantly, discusses the use of generative algorithms by one political agent in particular, providing some examples of the potential risks to society. This latter is of grave concern, and monitoring tools must be established. This paper presents a preliminary step to selecting relevant sources and setting a foundation for characterizing the mal-info, which must be addressed to ensure societal well-being. The AI/ML methods demonstrate a means for semi-quantitative signature capture. The impending use of "mal-GenAI" is presented. The main findings are: (1) we introduce specific politicized social channels of note; (2) we present viable indicative AI/ML modes of characterizing the output of these channels for analyzing and tracking mal-info; (3) we outline a case study with qualitative and semi-quantitative signatures of mal-info; (4) we flag the impending use of mal-GenAI in this context. It is critically important to develop mitigation strategies for mal-info, given the potential for societal harm.