Multimodal Contrastive Learning for <scp>3D</scp> Object Classification and Part‐Segmentation by Leveraging V‐<scp>LLM</scp> and <scp>CNNs</scp>

ABSTRACT Visual–Linguistic large models (V‐LLMs) have demonstrated strong semantic understanding capabilities in understanding visual open‐world scenes. However, there is limited research on how to effectively integrate these models with traditional deep learning networks for classify and segment 3D point clouds. This paper explores the joint training mechanisms of V‐VLLMs and Convolutional Neural Networks (CNNs), constructing a pluggable module that enhances 3D object classification and component segmentation using 2D images, 3D point clouds, and language descriptions. The idea of this pluggable module is to employ contrastive learning between the generalized features extracted by large models from text and image features extracted by the CNN to enhance the robustness and understanding accuracy of the original CNN‐based point cloud network. For experiments, we validated the effectiveness and accuracy of the proposed pluggable module using multiple point cloud classification frameworks, based on datasets such as ModelNet40, ScanObjectNN, and ShapeNetCore. The experimental results demonstrate that the multimodal contrastive learning pluggable module significantly enhances performance across various deep learning frameworks, resulting in significant enhancements in classification and segmentation precision. Additionally, for the multiscale feature PointNet++ framework, the proposed multiscale contrastive learning mechanism provides further performance gains. The Code is available at https://github.com/winbigqian/MCL‐3D‐VCNN .