Enhancing Transparency and Research Ethics through Human AI Techniques

In recent years, the empirical sciences have confronted a crisis of confidence following high-profile replication projects reporting disappointing results. The 'replication crisis' has led to widespread introspection and calls for reform across various fields. At the heart of this movement is the fundamental belief that reproducing, replicating, and interrogating published claims is essential to science. To facilitate this, many in the scientific community have emphasized the importance of openness and transparency in research, well-aligned incentives for researchers, robust peer review, and improved methods for meta-analyses over scientific corpora. Our work identifies with this movement, and in particular, highlights opportunities for emerging technologies to contribute to novel solutions in this space. This research proposes a coherent research agenda exploring artificial intelligence (AI) and hybrid human-AI technologies to support the evaluation of the replicability of scientific findings and understanding of science as a set of social and cultural processes. We discuss work developing and testing a new class of AI algorithms, specifically, artificial prediction markets populated by algorithmic traders who buy and sell contracts representing the outcomes of replication studies of published research outcomes. Given the critical role of human intuition and experience in making it more trustworthy, we explore ways in which artificial prediction markets can be modified to include human participants, so-called hybrid prediction markets. We study these hybrid systems both in simulation and in pilot testing with human participants in a real-world scenario. In a series of studies, we engage in extensive surveys and interviews with researchers to better understand the usefulness and design requirements for both AI and hybrid human-AI technologies in the scientific workflow. We solicit feedback on artificial and hybrid markets for estimating the replicability of published claims and focus on the role of transparency and explainability in evaluating scientific integrity. We aim to situate these concerns within social and cultural contexts, making explicit the norms and incentives driving current research ecosystems.