A Dataset for Addressing Patient’s Information Needs related to Clinical Course of Hospitalization

Patient's unique information needs about their hospitalization can be addressed using clinical evidence from electronic health records (EHRs) and artificial intelligence (AI). However, robust datasets to assess the factuality and relevance of AI-generated responses are lacking and, to our knowledge, none capture patient information needs in the context of their EHRs. To address this gap, we introduce ArchEHR-QA, an expert-annotated dataset of 134 cases from intensive care unit and emergency department settings. Cases comprise patient questions from public health forums, clinician-interpreted versions, relevant clinical note excerpts with sentence-level relevance annotations, and clinician-authored answers. To establish benchmarks for grounded EHR question answering (QA), we evaluated three open-weight large language models (Llama 4, Llama 3, and Mixtral) across three prompting strategies. We assessed performance on two dimensions: Factuality (overlap between cited and ground truth evidence) and Relevance (similarity to reference answers).