A Tool for Researching how AI Affects Information Seeking

Information Scholars are calling for more standardization of the metrics quantifying the information-seeking processes (ISP).Standardizing measures for online searching is challenging due to search pages including the dynamic nature of generative AI adjacent to personalized search results.AI-chatbots occasionally use queries as prompts to generate text using large-language models (LLM).Search engines also occasionally include Retrieval-Augmented Generation (RAG) results adjacent to personalized search results.This interface hinders the level of control in experiments when two similar queries are treated as prompts towards very different results and text-generations.In addition, experiment findings are less comparable when researchers use different combinations of software for data log collections (e.g., click-throughs and time-on-site), analysis (e.g., NVivo and Qualtrics), and visualizations.These different combinations of platforms can lead to different statistical results or even obnubilate the research metrics.To address these discrepancies, this study introduces a method with three key automations to assist with standardizing ISP experiments on search pages with LLMs and RAG: 1) Controls: A system designed for Randomized Control Trial Experiments and A/B Tests by simulating Google's search AI and algorithms; 2) Data Collection: Automatic participant log data collection; 3) Analysis and Visualization: Presentation of statistically significant differences in both quantitative and qualitative data, with results visualized alongside proper formatting (e.g., APA citations of the p-value).Preliminary feedback from Information Scholars has been promising, with 86% expressing sufficient value in the proposed method to consider using the software for their future projects.