Exploiting GPT-4 for Multimodal Medical Data Processing in Electronic Health Record Systems

The growing intricacy of medical data presents a significant challenge for electronic health record (EHR) systems, which are required to integrate multimodal data in order to enhance diagnostic precision and treatment efficacy. This paper proposes a novel GPT-4-based approach to enhance medical data processing capabilities in EHR systems. In particular, the GPT-4 was initially employed to encode clinical text data, and a pre-training and fine-tuning strategy was devised based on domain knowledge with the objective of enhancing its comprehension of medical terminology and context in accordance with the specific requirements of the medical field. Secondly, in order to address the issue of incomplete text data in EHRs, we propose a context-based adaptive filling strategy to facilitate the dynamic completion of missing information and enhance the integrity of text data through the analysis of analogous historical records. In order to enhance the model's capacity to comprehend lengthy texts, we have employed a hierarchical attention mechanism to partition lengthy texts into multiple sub-blocks. Through the adjustment of hierarchical attention weights, GPT-4 is capable of effectively capturing cross-paragraph and cross-topic relationships. Furthermore, data augmentation technology has been employed to generate diverse training data through semantic extensions and data augmentation models, thereby enhancing GPT-4's adaptability to diverse inputs. The experimental results demonstrate that the proposed method has led to notable enhancements in prediction accuracy, text generation capability, and data processing efficiency.