Enhancing Egocentric Insights: Comparison of Deep Learning Models for Food and Beverage Object Detection from Egocentric Images

The relationship between a balanced diet and human health is well-established. The emergence of wearable egocentric vision, coupled with automatic nutrition estimation powered by deep learning and the Internet of Things (IoT), holds great potential for advancing dietary monitoring. This paper conducts a comprehensive comparison of state-of-the-art deep learning models for food and beverage object detection from egocentric images. Egocentric image data were collected from 84 participants in a free-living environment using Automatic Ingestion Monitor Version-2 (AIM-2). Object detection models (YOLOv5, YOLOv8, RT-DETR, DINO, and Co-DETR) were fine-tuned on 48274 food and beverage images from randomly selected 74 participants, validated on 568 images (from 3 randomly selected participants) and tested on 3958 images (from rest of 7 participants). The results were evaluated by standard metrics (Pascal VOC [email protected], COCO [email protected]:0.95 or AP), and inference time. YOLOv5 achieved the highest [email protected] of 0.82 and AP of 0.46, while RT-DETR exhibited the fastest inference time (5.7 milliseconds). These results provide guidance in selecting the most suitable deep learning approach to enhance the accuracy and applicability of egocentric vision for automated dietary analysis through IoT.