Zero-Shot SAM for Food Image Segmentation

Recent advances in foundation models have enabled strong zero-shot performance across vision tasks, yet their effectiveness for fine-grained domains such as food image segmentation remains underexplored. This study investigates the zero-shot capabilities of the Segment Anything Model (SAM) for segmenting food images. We evaluate two prompting strategies—bounding boxes and coordinate points—and compare SAM’s results against established deep learning baselines, U-Net and DeepLabv3+, both trained on the FoodBin-17k dataset with pixel-wise binary annotations. Using bounding box prompts, zero-shot SAM achieved a Dice coefficient of 0.790, closely matching U-Net (0.759), while DeepLabv3+ attained 0.918. Coordinate-based prompting yielded comparable results, with the optimal configuration (75 coordinate points) also reaching a Dice coefficient of 0.790. These results highlight SAM’s strong potential for generalizing to novel visual categories without task-specific retraining. This study further provides insights into how prompt type influences segmentation quality, offering practical guidance for applying SAM in food-related and other fine-grained segmentation applications.