RedundancyLens: Revealing and Exploiting Visual Token Processing Redundancy for Efficient Decoder-Only MLLMs

Current Multimodal Large Language Model (MLLM) architectures face a critical tradeoff between performance and efficiency: decoderonly architectures achieve higher performance but lower efficiency, while cross-attentionbased architectures offer greater efficiency but lower performance.The key distinction lies in how visual tokens are processed.Decoder-only architectures apply self-attention and FFN operations on visual tokens, while cross-attention architectures skip these computations.To investigate whether redundancy exists in this computationally expensive process, we propose a training-free framework for analyzing trained MLLMs.It consists of Probe-Activated Dynamic FFN and Hollow Attention, which enable adjustable reductions in computations for visual tokens, as well as a Layer Ranking Algorithm that prioritizes layers for these reductions.Extensive experiments demonstrate substantial, structured, and clustered redundancy unique to decoder-only MLLMs, offering valuable insights for future MLLM architecture design.Furthermore, by leveraging our reduction framework as a training-free inference acceleration approach, we achieve performance comparable to or better than stateof-the-art methods while remaining compatible with them.