Navigating the AI Maze: Evaluating Emerging Tools in Academic Databases

Library database vendors have recently begun introducing AI tools to their platforms at an increasing pace. While some of these tools cannot be turned off, some can, leaving libraries to make the decision with little in the way of past practice to rely on. There has been considerable research done on the generally available large language models (LLMs) but less on methodologies for evaluating platform add-on tools. This paper outlines the development of such an evaluation process at an R2 academic library in the US, from conception to implementation. It offers practical insights on bringing in stakeholders across the library and recognizing their expertise while also being cognizant that this is added work. The complete evaluation tool is provided for readers and outcomes of three initial evaluations are discussed, along with ongoing concerns and vendor relations. The article is tightly focused on practical applications of the tool and advice for librarians. Readers will find themselves with a tool they can implement in their own institutions and background on its development to aid in scaling it up or down and modifying it to suit their own institutional needs. This will help ensure that library platforms can best meet the needs of their patrons.