ChatGPT-4 vs. Google Bard: Which Chatbot Better Understands the Italian Legislative Framework for Worker Health and Safety?

Large language models, such as ChatGPT-4 and Google Bard, have demonstrated potential in healthcare. This study explores their utility in occupational medicine, a field where decisions rely on compliance with specific workplace health and safety regulations. A dataset of questions encompassing key occupational health topics derived from the Italian Legislative Decree 81/08, which governs workplace health and safety, was utilized. Responses from ChatGPT-4 with contextual information (ChatGPT-4+context) and Google Bard were evaluated for accuracy and completeness, with error categorization used to identify common issues. Subcategories of the topics of the regulations were analyzed as well. In total, 433 questions were included in our analysis. ChatGPT-4+context surpasses Bard in terms of accuracy and completeness in responses, with a lower error rate in the categories analyzed, except for the percentage of missed responses. In the subcategories analyzed, Bard is superior to ChatGPT-4+context only in the areas of the manual handling of loads and physical hazards. ChatGPT-4+context outperformed Bard in providing answers about Italian regulations on health and safety at work. This study highlights the potential and limitations of large language models as decision-support tools in occupational medicine and underscores the importance of regulatory context in enhancing their reliability.