Natural-language cohort queries on a hospital data platform: accuracy and provenance in a real-world evaluation

Background: Accessing routinely collected clinical data for research remains technically demanding across heterogeneous hospital systems. This proof-of-concept study aimed to evaluate the accuracy and query-level traceability of a commercial natural-language (NL) data-query platform implemented at a university hospital. Methods: An on-premises data integration platform with an NL query interface (AIRE; Briya, Tel Aviv, Israel) was connected to the institutional health data platform across three tertiary-care hospitals under an approved pseudonymization and governance framework. Implementation comprised regulatory and governance approval, and IT deployment. Twenty-five NL prompts addressing cohort identification and characterization in an adult ICD-10 C71-coded brain-tumor cohort (n = 1,594) were evaluated against manually written reference SQL and checked for a corresponding executed query in the exported log. Responses were classified into five categories. Prompts with unanswered or incorrect responses were re-posed in a new thread. Results: The platform was implemented over two years. Fifteen of 25 responses were exact (n = 11), concordant after definition alignment (n = 1), or incorrect but traceable from the logged SQL (n = 3); six were plausible but incorrect without a logged query, and four remained unanswered. No incorrect unlogged result recurred in a new thread. Across the initial and repeat evaluations, 22 of 23 logged responses were correct or traceable, whereas all eight unlogged responses were incorrect. Conclusions: Implementation was feasible but required substantial local institutional preparation. The NL interface reproduced SQL-derived results when definitions were explicit, while query-level provenance and verification of the displayed result were critical for identifying unreliable outputs.