Multimodal autoregressive U-Net transformer architecture for pixelwise image and text conditioning (MUT)

The integration of multimodal data, combining both textual and visual information in a single context, has become increasingly important in many applications involving transformers. However, most existing models primarily focus on multimodal input, generating only textual outputs. This paper presents the Multimodal Autoregressive U-Net Transformer (MUT), a novel architecture designed to enable true multimodality by processing and generating both textual and visual tokens within the same context. By leveraging autoregressive large language models (LLMs) and incorporating a U-Net-inspired encoder-decoder for multiscale image processing, MUT seamlessly integrates and generates multimodal outputs. Experiments on the COCO2017 dataset demonstrate the model's ability to segment and reconstruct images based on textual prompts without requiring postprocessing steps. Despite some limitations in handling out-ofdomain images, the MUT architecture shows significant potential for tasks that require close integration of visual and textual modalities.