Knowledge and Task-Driven Multimodal Adaptive Transfer Through LLMs with Limited Data

The scarcity of annotated data and the challenge of generative models that meet the authenticity requirements collectively constrain the modeling of highly specialized deep medical models, making data remain a central issue in model training. The emergence of large-scale models has facilitated access to knowledge and data references for various tasks, even those with high specialization. Therefore, in the absence of data sources, latent knowledge and model structure can be considered for reuse instead of directly utilizing samples, and generalizing specific medical models based on large-scale models is expected to be competitive in the future. However, due to significant differences between fine-grained medical features and tasks, generalization based on large language models (LLMs) is not entirely satisfactory, it requires a more targeted inheritance of knowledge and important structures to achieve effective generalization. This paper presents a novel multi-modal medical model training approach designed to circumvent the need for extensive real data preparation, leveraging the capabilities of LLMs to generalize downstream tasks. Initially, knowledge threads are extracted to facilitate multi-modal diagnostic knowledge retrieval from LLMs without relying on massive datasets. These threads are then utilized to generate multimodal task instructions, guiding LLMs in providing feedback and further fine-tuning them for direct adaptation to downstream tasks by encoding samples into a unified feature representation, thereby simplifying the search and traversal process. Finally, an adaptive knowledge transfer strategy is proposed, implementing collaborative modeling with key patterns anchored under the joint influence of knowledge threads and task instructions, leading to the generation of an equivalent and concise initialization model guided by LLMs. The experimental design validates the efficacy of the approach in rapid generalizing specific models, demonstrating high accuracy and minimal resource consumption.