Chatbot Assessment: Best Practices for Artificial Intelligence in the Library

abstract: In November 2019, the Leonard Lief Library implemented Ivy.ai, a proprietary chatbot on its website. This implementation was the first academic library installation of a vendor-supplied chatbot to be discussed in the professional literature. This chatbot functioned as a new tool that assisted users seeking information from the library website. User questions provided insight to the authors about the kinds of topics students searched for via the library website. In April 2023, the chatbot’s vendor began using OpenAI’s ChatGPT Application Programming Interface (API) to improve the chatbot’s functionality. This change, from a rules-based chatbot system to a transformer model, enhanced the chatbot’s ability to provide answers to patrons. To better understand this major change, the authors assessed the chatbot’s usage during the Spring 2023 semester. This assessment revealed the kinds of questions the chatbot struggled to answer, and possible reasons why. The assessment’s findings demonstrated how chatbots can successfully function as a enhancement to the library website. The article also presents best practices for libraries looking to implement or experiment with chatbots and contributes to the ongoing discussion of artificial intelligence in libraries.