Generative AI as chemist’s virtual assistant: a case study with the chemistry of SmI2

The scientific community is standing on the verge of the emergence of artificial intelligence (AI) in chemical education and research. While the increasing role of AI-based technologies in research is inevitable, it is of paramount importance to verify and improve the output of AI-powered generative systems. We critically evaluated ChatGPT and Google Gemini’s (formerly known as BARD) performance on domain-specific chemical questions, leveraging our expertise in SmI₂ chemistry to benchmark the accuracy and conceptual alignment of AI-generated outputs with established literature. A series of questions related to the well-established chemistry as well as ongoing research of SmI2 were asked to ChatGPT and Google Gemini. The responses obtained from ChatGPT and Google Gemini were analyzed and correlated based on existing literature proof of SmI2 chemistry. Our analysis indicates that ChatGPT and Google Gemini can accurately answer many of the questions related to SmI2 chemistry. However, it has also been observed that these AI-based Chatbots provide completely or slightly different answers when asked multiple times or from different accounts. It was also observed that the response of AI Chatbots can be improved through introducing prompts in the questions. It is our observation and recommendation that these AI-based Chatbots are still a long way from replacing humans as scientific assistants.