A Comparative Study of Model Fine-tuning Methods for Multimodal Data Alignment of Electronic Products

Currently, after-sales Q&A systems for electronic products have replaced a significant amount of manual services, becoming essential for guiding after-sales maintenance and support. However, the after-sales service process faces challenges due to the diverse graphic and textual data associated with various product categories. Existing Q&A systems often struggle to align these datas effectively, making it difficult to accurately identify the attributes of the products needing service and their specific requirements. While the mainstream multimodal large model, CLIP, can effectively extract features from images and text and align graphical data, it lacks specialized domain knowledge, which hampers its ability to meet specific needs. To address this issue, this paper introduces relevant professional knowledge through model fine-tuning. Four fine-tuning approaches are employed: full parameter fine-tuning, partial parameter finetuning, added parameter fine-tuning, and LoRA (Low-Rank Adaptation) fine-tuning. These methods are trained using three labels: class, brand, and model, with HITS@K and MRR(Mean Reciprocal Rank) serving as performance metrics. Experimental results indicate that while the CLIP model demonstrates strong zero-shot learning capabilities, its performance in recognizing graphics within specialized domains is lacking. However, the four fine-tuning methods significantly enhance graphic alignment, with LoRA fine-tuning yielding the best results-approximately a 475 percent improvement compared to the pre-fine-tuning state. Additionally, ablation experiments on the LoRA method reveal that replacing the encoded part of the image with LoRA can have a substantial impact on the model’s performance.