Cluster-LLaVA: Mixture of Experts Multimodal LLMs Can Also be Small

This study proposes Cluster-LLaVA, a parameter efficient MoE multimodal large language models (MLLMs) structure. By leveraging the inherent activation sparsity of LLaVA and employing neuron clustering along with a multi-stage training strategy, Cluster-LLaVA reduces total parameters and active parameters demand while preserving performance. Experimental results show that it contains <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$50 \%-60 \%$</tex> of the parameters and activates <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$60 \%-70 \%$</tex> of the parameters, achieving over 95% of the baselines' performance.