Automated triaging of hospital complaints using Large Language Model-Assisted Content Analysis and machine learning

Hospital complaint management systems rely on manual review lacking standardised severity assessment, resulting in inconsistent triage. Automated Artificial Intelligence (AI)-based approaches could improve processing of high-volume unstructured feedback. To develop and evaluate an automated complaint classification system integrating Large Language Model-Assisted Content Analysis (LACA) with supervised Machine Learning (ML) for categorising hospital reviews by alert level. We analysed 1279 hospital reviews from Selangor, Malaysia (2023) using OpenAI's Generative Pre-trained Transformer (GPT)-4o-mini for thematic coding across 41 complaint codes on a five-point severity scale (Cohen's κ = 0.81, n = 200). After excluding neutral rating-3 reviews, 1181 reviews were modelled with Logistic Regression (LR) and Artificial Neural Network (ANN) using stratified splitting (80:20, n_test = 237) and random oversampling. The ANN comprised four hidden layers with Rectified Linear Unit (ReLU) activation, L2 weight decay, and Adam (Adaptive Moment Estimation) optimizer. An explanatory model combined factor analysis with logistic regression. Metrics included F1-score, accuracy, Area Under the Receiver Operating Characteristic Curve (AUC-ROC), and Brier score. LR achieved accuracy of 0.90 (high-alert F1 = 0.93, AUC-ROC = 0.965) and ANN achieved 0.85 (high-alert F1 = 0.90, AUC-ROC = 0.954). Factor analysis identified six thematic dimensions (Cumulative Explained Variance = 0.74, mean Cronbach's α = 0.86). Explanatory logistic regression (pseudo-R 2 = 0.55) identified Service and Communication Effectiveness as the strongest predictor (β = −2.1). LR is recommended for its superior performance and interpretability. The LACA–ML pipeline demonstrates competitive automated hospital complaint classification, offering a scalable and interpretable AI-driven approach for healthcare quality monitoring.