Applications of Large Language Models and Prompt Optimization for Knowledge Extraction From Biological Pathway Figures

Recent developments in Large Language Models (LLMs) have demonstrated remarkable capabilities for image comprehension. This study aims to automate and enhance the extraction of gene interactions from biological pathway images by integrating LLMs and a Genetic Algorithm (GA). A dataset of 200 tumor signaling pathway figures from the recent biological literature was employed to assess the performance of four AI chatbots: GPT-4oV, Claude-3.5V, Gemini-1.5V, and Llama-3.2V, with GA used to optimize prompts for each model. Model performance was evaluated on both directional and non-directional gene relationship extraction. GA-optimized prompts significantly improved extraction accuracies across all LLMs, with GPT 4oV achieving an F1-score of 0.645 (±0.055) and Llama-3.2V achieving an F1-score of 0.616 (±0.068). For non-directional interactions, GPT-4oV outperformed other models, reaching a precision of 0.805, a recall of 0.695, and an F1 score of 0.757, followed by Llama-3.2V and Claude-3.5V with F1-scores of 0.702 and 0.697, respectively, while Gemini-1.5V lagged with 0.612. In directional interaction predictions, all models performed lower, with GPT-4oV leading at 0.687 F1-score, followed by Llama-3.2V at 0.656, Claude-3.5V at 0.641, and Gemini-1.5V at 0.573. While these results demonstrate substantial improvements over traditional OCR-based approaches, further advances in model accuracy and explainability are needed for widespread adoption in critical biomedical applications. Nevertheless, these findings provide a valuable benchmark for the research community and a foundation for future development of specialized, fine-tuned models and scalable multimodal AI frameworks in biomedical data analysis. The source code is publicly available on https://github.com/Muh-aza/LLM_GPV.