Understanding Large Language Model Driven Social Bots: A Behavioral Analysis and Impact Assessment

As the capabilities of large language models (LLMs) emerge, they not only assist in accomplishing traditional tasks within more efficient paradigms but also stimulate the evolution of social bots. Researchers have begun exploring implementation of LLMs as the driving-core of social bots, enabling more efficient and user-friendly completion of tasks such as social behavior decision-making and social content generation. However, there is currently a lack of systematic research on behavioral characteristics of LLMs-driven social bots and their negative impact on social networks. We have curated data from <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Chirper.ai</i>, a Twitter-like social network populated by LLMs-driven social bots and embarked on an exploratory study. Our findings indicate that: 1) LLMs-driven social bots possess enhanced individual-level camouflage while exhibiting certain collective characteristics; 2) these bots have the ability to exert influence on online communities through toxic behaviors; and 3) existing detection methods are applicable to LLMs-driven social bots but may have certain limitations in effectiveness. Moreover, we organized the data collected in our study into Masquerade-23 dataset, which we have publicly released, thus addressing the data void in subfield of LLMs-driven social bots behavior datasets. Our research outcomes provide primary insights for the research and governance of LLMs-driven social bots within the research community.