Disciplinary Perspectives on the Relevance of AI Research Assistants for Scholarly Article Discovery

This study explored artificial intelligence (AI) research assistants through the lens of scholars from different disciplines as they interacted with the tools. The goal was to determine whether AI tools provide relevant scholarly literature or prioritize surface-level relevance at the expense of disciplinary depth, and to examine how scholars perceive the usefulness of the tools in teaching and learning. The study used a multiple-phase, qualitative design in which ten scholars from several disciplines completed structured search tasks in each AI research assistant (Consensus, Elicit, SciSpace), followed by semi-structured interviews. Data were analyzed using qualitative coding techniques, with themes developed through thematic analysis approach. Analysis involved data familiarization, memoing, iterative first and second cycle coding, and the generation and refinement of themes. The findings revealed six themes from the search task diaries and interviews with participants: article coverage varied across disciplines; discipline specific information needs and resource mismatch; a complementary resource for scholarly article discovery; AI summaries technically correct but generic; faculty perceptions on AI tools in student learning; usability and cognitive load considerations. The findings indicate that while most tools surfaced additional research from scholarly journals, they often did not consistently identify seminal literature across the sampled disciplines. AI summaries on the literature were perceived as factually correct but generic. Lastly, participants expressed varied perceptions about the ways in which the tools could be used in teaching and learning contexts.