Retrieval augmented generation for smart calorie estimation in complex food scenarios

Accurate food recognition and calorie estimation are critical for managing diet-related health issues such as obesity and diabetes. Traditional food logging methods rely on manual input, leading to inaccurate nutritional records. Although recent advances in computer vision and deep learning offer automated solutions, existing models struggle with generalizability due to homogeneous datasets and limited representation of complex cuisines like Indian food. This paper introduces a dataset containing over 15,000 images of 56 popular Indian food items. Curated from diverse sources, including social media and real-world photography, the dataset aims to capture the complexity of Indian meals, where multiple food items often appear together in a single image. This ensures greater lighting, presentation, and image quality variability compared to existing data sets. We evaluated the data set with various YOLO-based models, including YOLOv5 through YOLOv12, and enhanced the backbone with omniscale feature learning from OSNet, improving detection accuracy. In addition, we integrate a Retrieval-Augmented-Generation (RAG) module with YOLO, which refines food identification by associating fine-grained food categories with nutritional information, ingredients, and recipes. Our approach demonstrates improved performance in recognising complex meals. It addresses key challenges in food recognition, offering a scalable solution for accurate calorie estimation, especially for culturally diverse cuisines like Indian food. • We propose a CAM-enhanced YOLO architecture that improves detection sensitivity for small and fine-grained food items in complex meal settings. • A large-scale Indian food dataset with balanced classes, designed to overcome the scale, diversity, and imbalance issues of existing datasets. • A RAG-based calorie estimation pipeline that connects detection outputs to nutritional databases, enabling accurate and automated calorie tracking. • We present an extensive evaluation of existing datasets alongside our proposed dataset using various YOLO models, incorporating both SPPF and CAM modalities. • We further evaluate the pipeline for retrieval accuracy and text matching to assess the RAG model’s output against the augmented class query.