Evaluation and Integration of Advanced AI Chatbots for Biological Pathway Curation

The rapid expansion of biological literature presents significant challenges in manually curating pathway knowledge from images for biological and medical research. Recent advancements in AI, particularly multimodal AI chatbots like GPT-4V, Claude, and Gemini, offer promising capabilities for image understanding and extraction of biological interactions. This study evaluates the effectiveness of these AI chatbots in curating gene interactions from 50 curated tumor signaling pathway images. Our results highlight the superior performance of GPT-4V, achieving a precision of 0.778, a recall of 0.664, an F1-score of 0.716 in non-directional analysis, and a precision of 0.566, a recall of 0.471, an F1-score of 0.512 in directional analysis. Claude followed with a precision of 0.735, recall of 0.594, and F1-score of 0.6571 in non-directional analysis, and a precision of 0.500, recall of 0.400, and F1-score of 0.445 in directional analysis. Gemini recorded the lowest performance metrics across both analyses. We further developed a model fusion integrating Claude-3 chatbots and our pathway curation pipeline, achieving notable improvements.