A Process for Building Datasets that Enable the Application of Different Methods for Detecting Fake News and Social Bots

Context: The spread of fake news on social media is an imperative concern. The dissemination of such news by social bots has added complexity to disinformation detection applications. Machine learning methods have been applied to classify news as fake and not fake and accounts as bot and not bot, based on labeled datasets. Problem: There are no datasets that embed simultaneously news labeled as fake and not fake and accounts labeled as bot and not bot. This gap hinders the evaluation of classification methods that could benefit from such data embedding. Solution: A process for building datasets that contain pieces of news and accounts appropriately labeled, and enable development and comparison of fake news and social bots detection methods. IS Theory: General Systems Theory1 and social Network Theory2. Method: Data requirements from SOTA3 fake news and social bot detection methods guided the development of the process. This process collects data from social networks and fact-checking agencies. A case study generated a dataset, illustrating the viability of the process. Summary of Results: The dataset generated is public and contains 440 labeled pieces of news and 6,274 labeled accounts. Most fake news detection methods improved their performance when they considered the labels of the accounts. Contributions and Impact on the IS field: The process that builds datasets that integrate labeled news and labeled accounts, and the dataset generated by the case study. Both contributions are related to the Grand Challenges in IS Research and the Sociotechnical Vision of IS.