Development and preliminary usability evaluation of an evidence-based knowledge graph–supported intelligent question-answering system for chronic rhinosinusitis health education

Background Chronic rhinosinusitis (CRS) affects 4.8%~12.0% of the world's population and seriously reduces the quality of life of patients. Traditional CRS health education is limited by the time of medical staff and the quality of online health education content is uneven. Objective To develop an evidence-based knowledge graph–supported intelligent question-answering system for chronic rhinosinusitis health education and to preliminarily evaluate its functional performance, usability, and user-reported satisfaction among nurses and patients. Methods An evidence-based CRS health education knowledge base was developed through literature retrieval, methodological quality appraisal, evidence synthesis, and expert consultation. The finalized content was manually annotated and transformed into machine-readable triples, and the knowledge graph was visualized using NRD Studio. An intelligent question-answering function was developed using the Coze platform and integrated with the knowledge base for retrieval-augmented response generation. The system was made accessible to users through a WeChat official account, which served as the unified mobile access point for both the intelligent question-answering function and the knowledge graph visualization. Before user evaluation, the system was tested using 30 standardized questions covering six domains as a development set. Following refinement of the knowledge-base content and prompts in response to seven identified issues, the system was retested on the same question set 2 days later. Responses were independently evaluated by clinical nurses for content consistency, safety, and appropriate refusal of out-of-scope questions. Subsequently, 15 clinical nurses and 15 hospitalized patients with CRS were recruited through convenience sampling and used the system for 1 week. Usability was assessed using the System Usability Scale, and patient-reported health education satisfaction was assessed using a health education satisfaction questionnaire. Results The knowledge graph consisted of 287 nodes, 42 relationship types, and 419 inter-node links. During the initial functional test on the development set, seven of the 30 standardized questions were identified as requiring refinement because of incomplete information or inappropriate wording. After modification of the corresponding knowledge-base content and prompts, the system was retested on the same development set 2 days later. On the development set, the two evaluators recorded scores of 1 for both content consistency and the study-specific safety criterion for all 25 in-scope questions, while all five out-of-scope questions were recorded as appropriately refused. These ratings do not establish independent clinical safety or validate the subsequently revised reference answers. Inter-rater agreement was 100% for all binary ratings; because all ratings were identical, Cohen's kappa was not estimable, and PABAK was 1.00. These results represent development-set performance and should be interpreted as preliminary findings; they do not establish a general accuracy or safety rate. Among user participants, the mean System Usability Scale scores were 90.33 ± 7.31 (95% CI 86.28–94.38) for nurses and 85.00 ± 4.91 (95% CI 82.29–87.72) for patients. Patients reported a mean health education satisfaction score of 68.53 ± 2.42 (95% CI 67.19–69.87) out of 72. Conclusions The evidence-based CRS knowledge graph–supported intelligent question-answering system demonstrated favorable usability, while formative functional testing on a development set yielded favorable evaluator ratings under the study-specific content-consistency and safety criteria. The system may serve as a supplementary digital tool for CRS health education; however, these findings should be interpreted as preliminary development-set results and require validation on independent test sets before broader claims about accuracy and safety can be made.