Cuidado materno: ¿es bueno confiar en recomendaciones de IA que usan guías clínicas?

Introduction. Medical staff have limitations when consulting and applying clinical guidelines in practice. Recent AI technologies, such as Large Language Models (LLMs), can help to overcome these challenges. When Retrieval-Augmented Generation (RAG) is applied to LLMs, the outputs become more relevant in specific contexts and consistently adhere to medical guidelines.Objectives. To evaluate the performance of commercial LLMs, as measured by their answers, when using questions related to maternal health on RAG systems, using both human and automatic metrics.Material and methods. We designed a controlled experiment to obtain accurate, consistent answers from an RAG system using Colombian guidelines for maternal care. A physician formulated ten questions and defined the ground truth answers. We created a prompt and evaluated several LLMs using 1) Answer-Binary Human Concept ranking and 2) the Retrieval-Augmented Generation Assessment (RAGAS) metrics judged by another LLM.Results. The GPT-3.5 model achieved the highest score in physician evaluation with 0,9 accuracy. According to the assessment of the GPT-4o judge, the Claude 3.5 model was the highest rated with a faithfulness score of 0,78. Meanwhile, under the Claude 3.5 judge, the Mistral model ranked highest with a faithfulness of 0,84. Regarding answer relevancy, GPT 3.5 scored the highest for both judges, with 0,94 and 0,86.Conclusions. Integrating RAG in obstetrics can enhance evidence-based practices, improving patient outcomes. However, assessing answer accuracy and context-specific information before clinical use is crucial. Our results suggest using heavy models, such asGPT-3.5, Claude, or Llama 70B over lightweight models like Llama 8B.