From Pixels to Voice: A Simple and Efficient End-to-End Spoken Image Description Approach via Vision Codec Language Models

Neural audio codecs provide a powerful tool for compressing audio signals into discrete codec representations. This compact discrete representation has made it possible to successfully apply a natural language processing (NLP) model to various audio and speech processing tasks, including text-to-speech (e.g., VALL-E, VALL-E X) and multimodal audio-text generation (e.g., LauraGPT, VioLA). While these models excel at handling sequential data like text and speech, their potential for processing non-sequential data, such as images, remains unexplored. In this paper, we introduce PixVoxLM, a simple and efficient end-to-end framework that combines vision-language models with neural audio codecs to tackle the Image-to-Speech (I2S) problem. Experiments on the Flickr8k dataset demonstrate that PixVoxLM delivers promising results compared to existing I2S methods. Furthermore, this research is the first to explore a new capability: visual-guided speech completion in I2S model, paving the way for new practical applications in everyday communication, such as speech prompt-based instruction.