Blind-Assisted Question Answering Optimization Model Based on Llava

At present, many models focus on the field of blindassisted question answering. However, most models only stay at the level of relatively simple visual recognition. Multimodal visuallanguage models face the problems of high computational cost and insufficient real-time performance in blind-assisted question answering tasks. This situation limits its application on resourceconstrained devices. Based on this, an efficient question answering solution for LLaVA is proposed to optimize the performance of blind-assisted scenarios. This model integrates Gemma's efficient model into the LLaVA architecture, reducing the size of the language model, and replaces the feedforward network with a hybrid expert layer to achieve sparsity, thereby reducing the number of parameters. Given that visual perception tokens are generally several times that of text tokens, it is proposed to replace visual tokens with perception weights, which greatly reduces the inference speed and achieves lightweight deployment. Experiments show that the model maintains a certain accuracy on multiple common data sets and its efficiency is greatly improved, providing an efficient multimodal model for the field of blindassisted question answering.