Exploration of lightweight SAM + CLIP: harnessing the power of advanced image understanding and segmentation

Currently, many Natural Language (NL) to Image Recognition (IR) models are not fully automated, requiring large amounts of input commands from experts in image processing. Those that achieve full automation generate heavy workloads, placing immense burdens on hardware and systems. The time required to process these automated tasks is often long as well. To address this problem, this paper proposes an innovative way to combine the State-Of-The-Art (SOTA) Segment Anything Model (SAM) with Contrastive Language-Image Pretraining (CLIP), offering a new method to implement an NL2IR framework with relatively low workload and faster response times. This paper also aims to increase the accuracy rate of SAM when combined with CLIP, as well as explore possible fine-tuning approaches for the combined model.