Adaptive Human-LLMs Interaction Collaboration: Reinforcement Learning driven Vision-Language Models for Medical Report Generation

The large language models (LLMs) and vision models have demonstrated great potential in the field of healthcare. The integration of vision-language models (VLMs) has led to significant advancements in automatic medical report generation. However, effectively incorporating doctor feedback into the report generation process remains an unresolved challenge in clinical practice. In this paper, we propose an adaptive human-LLMs collaborative vision-language model that dynamically adjusts the report generation process based on the feedback of doctors and LLMs. We designed a reinforcement learning module to dynamically update the generated content generated by the VLMs according to doctor feedback, addressing the challenge of integrating doctor feedback into the generation model. To mitigate over-reliance on doctor feedback, we introduced the RL-LLM module, which uses an LLM to assess the quality of the generated report. The model adopts a participatory design approach in collaboration with clinical doctors to improve report accuracy and clinical availability.