Automated clinical decision support based on recognition of physiological attributes

Trauma resuscitation refers to the immediate medical response provided to individuals suffering from severe injuries or trauma. The primary objectives are to stabilize the patient’s condition, prevent further harm, and initiate life-saving medical interventions to improve survival outcomes. However, the inherently complex and multidisciplinary workflow in trauma resuscitation often leads to human delays and errors, which can compromise the quality of care delivered by medical teams. To address these challenges, data-driven Clinical Decision Support Systems (CDSS) have been proposed to assist clinical decision-making. Although CDSS has shown promise in reducing human errors and delays, its reliance on manual input of physiological attributes renders it impractical in urgent and chaotic trauma settings. In critical situations, medical providers may be unable to input necessary data, rendering CDSS ineffective. To overcome these limitations, automating the input process for CDSS is essential to enhance its utility in trauma resuscitation. This thesis presents the development of a three-phase system designed to automatically capture the required inputs for CDSS.Automatic Speech Recognition (ASR): A customized ASR model was developed to transcribe verbal communication between team members during trauma resuscitation. Due to the limited size of the trauma dataset—stemming from labor-intensive labeling and patient privacy concerns—we adopted a dual approach utilizing both hybrid (deep neural network–hidden Markov model [DNN-HMM]) and End-to-End (E2E) frameworks. The hybrid model was employed as a data-cleaning mechanism to filter out noisy samples, such as overlapping speech and background noise, which would otherwise corrupt E2E model training and lead to hallucination issues. Using the cleaned dataset, we trained the E2E model with advanced techniques, including self-supervised representation learning, to leverage large amounts of unlabeled data. Our ASR model achieved a word error rate (WER) of 46.2%, outperforming widely available ASR engines such as Whisper and Azure. While E2E models demonstrated relative robustness to noise (where speech enhancement could degrade performance), they were not inherently designed to handle simultaneous speech from multiple speakers, which is known as the cocktail party problem. To address this, we further explored speech separation techniques, proposing a novel training strategy called dynamic sample dropout (DSD). Combined with layer-wise optimization (LO), combining these methods significantly improved the training stability and convergence speed of existing speech separation models.Multimodal Contextual Activity Recognition: We proposed a multimodal transformer-based model to identify medical activities commonly performed during trauma resuscitation based on verbal communication. This model has several key features: (1) it aggregates complementary information from multiple modalities to improve recognition accuracy; (2) it employs a keyword spotting strategy, focusing on a finite set of critical keywords instead of the entire vocabulary, thereby mitigating the adverse effects of suboptimal ASR performance; and (3) it incorporates a context module that captures dependencies and patterns in team communications, providing a better understanding of trauma workflows. For offline evaluation, our proposed model achieved weighted F₁ scores of 73.2% and 78.1% for activity and activity-stage, respectively. For online evaluation, the model achieved 57.4% and 61.6% weighted F₁ scores for activity and activity-stage recognition. Our results demonstrated the effectiveness of proposed approach in accurately recognizing medical activities in real-time scenarios.Physiological Attributes Detection System (PADS): To meet the input requirements of CDSS, we developed the Physiological Attributes Detection System (PADS), which extracts key physiological attributes from the transcribed speech and recognized medical activities. Physiological attributes are numerical or descriptive indicators reflecting the status of medical procedures or patient conditions, our goal is to detect 7 attributes associated with primary survey during trauma resuscitation. To validate the performance of PADS in real-world settings, we collected and annotated 27 trauma cases from a Level-1 trauma bay between June and July 2023. The results showed that our proposed two-stage approach for detecting these achieved an accuracy around 72% for 7 attributes, and demonstrated that PADS is capable of capturing these attributes in real-time, showcasing its feasibility and potential for practical deployment in high-pressure trauma environments. The proposed three-phase system for automatic CDSS, including an ASR model, a multimodal activity recognition system, and a PADS, represents a significant advancement in automating input capture for CDSS during trauma resuscitation. By addressing challenges such as noisy environments, overlapping speech, and limited training data, our system enhances the accuracy and reliability of input data for CDSS. The integration of speech separation techniques, contextual modeling, and multimodal data fusion ensures robust performance in real-world scenarios. Preliminary evaluations confirm the feasibility and effectiveness of the system in improving the workflow and decision-making process in trauma care. These findings underscore the potential of leveraging automated technologies to minimize human errors, reduce delays, and ultimately improve patient outcomes in critical care settings.