Using Natural Language and Health Ontologies in Hope Recommender System: Evaluation of Use in Medicine

Objectives: Despite the widespread availability of digital clinical information, timely access to relevant biomedical evidence during routine consultations remains limited in practice. Primary care clinicians, in particular, face significant time constraints that make it difficult to integrate comprehensive literature searches into everyday workflows. This study evaluates whether an ontology-based recommender system can support routine clinical workflows by reducing information retrieval time while preserving the clinically acceptable usefulness of retrieved evidence. We assessed the performance of the HOPE (Health Operation for Personalised Evidence) system compared with realistic manual PubMed searches conducted by physicians. Materials and Methods: We conducted an observational evaluation involving 50 primary care physicians, who independently assessed 30 anonymised, rewritten clinical cases representative of common primary care scenarios. HOPE automatically extracted biomedical concepts from case descriptions using natural language processing and mapped them to Unified Medical Language System (UMLS) ontologies to generate ranked PubMed recommendations. A subset of 10 physicians also conducted manual PubMed searches in line with their usual clinical practice. Article relevance was assessed using a predefined binary criterion, and a reference relevance set was established by consensus among three senior physicians using a pooled document set. Retrieval performance was evaluated using Precision@k, relative Recall@k, and Normalised Discounted Cumulative Gain (NDCG@k). Manual search time was measured using a standardised stopwatch protocol, whereas HOPE response time was logged automatically by the system. Results: Inter-physician agreement in relevance assessment was substantial (Fleiss’ κ = 0.66; 95% CI: 0.61–0.70). HOPE achieved moderate-to-high precision within the top-ranked results (Precision@3 = 0.72), with relative recall increasing as additional documents were considered. Ranking metrics indicated that relevant articles were generally positioned early in the result lists. The mean total retrieval time for manual PubMed searches was 13.3 ± 1.7 min per case, compared with 17.4 ± 2.1 s for HOPE-assisted retrieval (p < 0.001). Conclusions: In a controlled, workflow-oriented evaluation using synthetic clinical cases, HOPE substantially reduced information retrieval time while maintaining clinically acceptable relevance in the retrieved literature. These findings support the use of ontology-based, AI-assisted systems as workflow-support tools to facilitate timely access to biomedical evidence, without replacing clinical judgment.