RGB-D food nutrient estimation supported by FLAVA contrastive learning

Accurate estimation of absolute nutrient values in food remains a significant challenge for automated dietary assessment. We propose a novel multi-modal deep learning framework that integrates RGB-D imaging with vision-language contrastive learning for food nutrient estimation. Our dual-pathway architecture employs Swin Transformer V2 Tiny backbones to process RGB and depth information separately, followed by hierarchical feature mixing across multiple scales to capture both fine-grained details and global food representations. We integrate the FLAVA (Foundational Language And Vision Alignment) model to enable vision-text contrastive learning, aligning visual features with ingredient descriptions to enhance semantic understanding of food composition. Additionally, we implement vision-vision contrastive learning to ensure consistency between different visual representations. Evaluated on the Nutrition5k dataset containing 3,490 RGB-D images with precise nutritional measurements, our approach achieves state-of-the-art performance with a mean Percentage Mean Absolute Error (PMAE) of 14.43% across all nutritional components, representing significant improvement over the previous best of 15.9%. FLAVA integration guides model training with ingredient information but is not employed during testing, significantly reducing computational demands. With only 0.44-second inference time, our approach is suitable for real-time applications including mobile deployment. While achieving state-of-the-art results on Nutrition5k, the model's performance on diverse global cuisines requires further validation. • Achieves state-of-the-art 14.43% mean PMAE on Nutrition5k • RGB-D + FLAVA contrastive learning framework for accurate food nutrient estimation • Ingredient-guided training improves accuracy without requiring text at inference