A Technical Framework for Recognizing and Interpreting Complex Medical Records: Based on Multimodal Large Language Model

This paper brings up a technical framework for Interpreting medical documentation that integrates multi-modal large language modals, aiming to provide patients and doctors with the ability to read non-standard medical documentation in complex environments and to obtain text interpretation based on generative AI. The framework proposes a three-stage solution, namely Recognition, Formatting, and AI-Processing, involving technologies such as OCR, medical multi-modal models, and large language models, abbreviated as the RF-AI framework. This paper focuses on explaining the potential issues encountered when applying the framework and describes the resolution within the framework. In addition, this paper conducts experiments on two key stages of the framework, recognition and AI-processing, which effectively demonstrate the feasibility of the framework. This framework can significantly reduce the difficulty for patients in understanding medical records and provides necessary resolution for situations where paper records might be used. It helps patients better understand their conditions and enhances the efficiency of diagnosis and treatment between doctors and patients. Benefiting from a large language model, this framework allows developers to expand based on actual needs and can be integrated into existing electronic healthcare systems to achieve more comprehensive functionality.