Evaluating generative AI tools in Arabic academic libraries: a comparative analysis of ChatGPT-4o and Gemini’s performance and practical queries

Purpose This study aims to evaluate the performance of artificial intelligence (AI) tools ChatGPT-4o and Gemini in providing support in Arabic libraries, analyzing their ability to potentially answer diverse queries and exploring the feasibility of using such technologies. Design/methodology/approach Descriptive analytical and content-analysis methods were adopted. A set of queries was directed to each tool individually, and their responses’ accuracy, comprehensiveness and presentation style were evaluated. Statistical analysis was conducted to compare the tools’ performance. Subject experts also evaluated the tools in their respective fields. Findings The results showed that Gemini significantly outperformed ChatGPT-4o in most question categories, providing more accurate and comprehensive responses. Gemini showed strength in social sciences, humanities and educational sciences. However, both tools had limitations with questions about current events and those requiring detailed local knowledge. Practical implications This study contributes empirical evidence on AI tools’ potential benefits and limitations in Arabic academic libraries. Information professionals should critically evaluate these technologies and consider the ethical implications of their use. Our results imply that Gemini can potentially enhance Arab researchers’ finding information, particularly in certain subject areas. Libraries should develop strategies for the responsible and effective implementation of AI tools. Originality/value This study enriches the literature on AI in libraries by focusing on the Arabic context and providing insights from a multifaceted analysis of two prominent AI tools.