Deep Learning Models Connecting Images and Text: A Primer for Radiologists

In radiology practice, medical images are described and interpreted by radiologists in text reports. Recent technical developments enabling deep learning models to connect images and text may facilitate the radiologic workflow. These developments include advances in data embedding, self-supervised learning, zero-shot learning, and transformer-based model architectures. Models connecting images and text can be divided into four categories: <i>(a)</i> Text-image alignment models associate text descriptions with corresponding images. <i>(b)</i> Image-to-text models create text descriptions from images. <i>(c)</i> Text-to-image models generate images from text descriptions. <i>(d)</i> Multimodal models integrate and interpret multiple types of data such as images, videos, text, and numbers simultaneously. Potential clinical applications of these models include automated captioning of medical images, generation of the preliminary radiology report, and creation of educational images. These advances may enable case prioritization, streamlining of clinical workflows, and improvements in diagnostic accuracy. Published under a CC BY 4.0 license.