From the Wild West to the Walled Garden

Introduction The proliferation of large online platforms has fundamentally reshaped societal processes, from political communication and civic interaction to the mundane organisation of everyday social life (Jungherr et al.). The large volumes of digital traces and interactions generated on these platforms offer researchers an unprecedented opportunity to study both contemporary society and how these digital environments shape it, enabling analysis at a scale and granularity that would otherwise be unattainable. However, accessing such data has been a persistent challenge, as a result of competing interests involving user privacy, platform commercial incentives, and the public’s right to scrutiny (Weller). The development of academic data access has been uneven, evolving from informal, ad hoc collaborations to the present more regulated era (Darius et al.). It is important to regard questions of data access in relation to large online platforms against the background of their outsized societal importance and the risks and benefits associated with their widespread use. The present policy initiatives that have transformed how researchers gain access to and work with platform data are a reaction to prior modes of working with such data, which were largely unregulated. A number of widely publicised scandals, such as the Cambridge Analytica (Hu) scandal and the OkCupid data breach, prompted public discussion about how to counter potential misuse, including in the context of scientific research (Puschmann and Bozdag). At the same time, because so many societal processes, political communication, and civic interactions take place through platforms, they are of key importance for a functioning democracy. Therefore, a middle ground between preserving user privacy and allowing researchers to analyse data that is of key importance for understanding society and safeguarding against systemic threats had to be found. This article traces the evolution of the technical and regulatory frameworks for accessing social media data for research purposes, with a particular focus on the trajectory of Twitter (now X). While we centre our analysis on Twitter/X, its trajectory is far from unique. Meta’s shuttering of CrowdTangle in 2024—a tool that had become indispensable for researchers studying the spread of content on Facebook and Instagram—removed a key transparency resource with little prior consultation of the research community (Bruns). Reddit’s controversial decision in 2023 to impose steep API pricing effectively ended many third-party research tools, while TikTok has never offered meaningful programmatic data access for researchers outside of limited partnerships. These parallel developments underscore that the story we trace here is not idiosyncratic to one platform but reflects a structural pattern across the platform economy. The history of Twitter serves as a powerful case study, illustrating the interplay between technological infrastructure, corporate strategy, and regulatory oversight. We provide a brief account of the early, unregulated “Wild West” of data access—a term used to describe the largely ungoverned landscape of social media research prior to major platform restrictions (Puschmann, “An End”)—, the rise and fall of platform-led initiatives, and the recent, transformative impact of comprehensive legislation like the European Union’s Digital Services Act (DSA). The closure of open APIs, which Bruns termed the “APIcalypse”, signalled a turning point in this trajectory. Ultimately, we argue that the path to data transparency is not a linear progression but uncertain terrain, where progress can be fragile and subject to the shifting priorities of powerful platform owners. Of Data Philanthropy and Ad Hoc Partnerships Prior to the late 2010s, the landscape of academia-industry collaboration was characterised by a model best described as “data philanthropy”. In this paradigm, platforms selectively shared data with researchers based on their own criteria, often through informal relationships or one-off initiatives. These early partnerships were not governed by standardised rules but by the goodwill and strategic interests of the companies themselves, leading to a system that was often criticised for its lack of transparency and potential for bias. This occurred either as the result of a formalised call for participation—often in reaction to a critical event, such as suspected election manipulation or the spread of hate or misinformation—or without a formalised call, as the result of mutual contact and shared academic pedigree (Darius et al.). Both variants were criticised as being intransparent and subject to biases and conflicts of interest on the part of platforms. Over time, the informal model, in which informal ties between industry researchers and academics—often on the basis of mutual acquaintance or shared academic pedigree—were the basis for collaboration, gave way to the more formalised variant in which a process was designed that resulted in data sharing based on a selection process of project applications. This shift over time mostly occurred as a reaction to public objections, as the collaborations built mostly on social ties and shared research interests were widely criticised. In some cases, these initiatives of company founders or senior management were viewed as favouring specific individual researchers. However, the model that replaced this, which involved some level of scrutiny through an open call, was still often regarded as quite intransparent because the selection criteria were not clear. This era was marked by several high-profile, yet selective, collaborations that highlighted both the promise and the peril of platform data. Facebook, for instance, engaged in deep collaborations with social scientists to conduct massive field experiments. A prominent example is the “61-million-person experiment” on social influence and political mobilisation, published in Nature in 2012, which manipulated the News Feeds of millions of users to test the effects of social messages on voter turnout (Bond et al.). A similar, more controversial initiative was the 2014 “Emotional Contagion” study, where the algorithm was manipulated for nearly 700,000 users to test whether emotions could spread through a social network (Kramer et al.). These projects, while yielding significant scientific insights, were relationship-based and not open to the broader research community, raising concerns about preferential access and the power of platforms to shape the research agenda. The model of “independence by permission”, where research operates within parameters set by the studied entity, became a recurring theme. Other platforms also experimented with structured data sharing. In 2014, Twitter launched its “Data Grants” program, a pilot initiative that provided a small number of selected research projects with free access to historical public tweet data, which were otherwise prohibitively expensive via resellers. The selected projects were highly issue-specific, with grants awarded to study influenza trends and social media use during flood disasters, marking one of the first times a major platform acknowledged the public utility of their data by formalising a donation mechanism for academia. Similarly, Microsoft Research collaborated with academics to use anonymised Bing search logs for public health research, such as identifying adverse drug reactions missed during clinical trials. Yahoo’s Webscope program, established in 2006, operated like a reference library, offering curated, static datasets—such as language models and user interaction graphs—to university faculty who signed data use agreements, providing a critical resource for information scientists who lacked access to the internal data of search engines. A case underscoring the limits of digital trace data sharing was the 2006 AOL Research search data release. In an attempt to support the academic community, AOL released a dataset of 20 million search queries from 650,000 users. Despite attempts at anonymisation, journalists at The New York Times were able to re-identify individuals based on their search histories. The incident reinforced a move away from open data releases and toward more controlled, selective access models, and underscored the profound risks to individual privacy. Evolution of Data Access on Twitter/X The history of Twitter as a data source for academic research provides a microcosm of the broader trends and reversals in the field. Its evolution can be divided into three distinct phases, tracing a path from openness to more formalised partnerships and, ultimately, to a precipitous retreat behind a commercial paywall. Phase 1: The ‘Wild West’ of the Open API (2006–2020) In its early years, Twitter was extremely popular as a source of social media data. Its public-by-default nature, combined with a robust and relatively permissive set of developer APIs, created a “Wild, Wild West” environment for data acquisition. The developer APIs were originally intended to foster a third-party ecosystem of clients and apps, but academics quickly discovered their utility for research. During this phase, a large body of scholarship emerged, characterised by experimentation and description, often only loosely grounded in theoretical assumptions. Researchers explored everything from breaking news and political debate to social movements and crisis communication. As researchers identified the platform’s strengths—its simplicity, large data volumes, and focus on public, real-time communication—access remained largely unregulated, and those with the technical skills could collect large datasets with relative ease. This period established Twitter as an invaluable, if chaotic, real-time sensor for social and political life. Phase 2: The ‘Golden Age’ of the Academic API (2020–2023) As the field matured and calls for more structured access grew, Twitter took a step t