ViT-Based Hybrid Segmentation for Leftover Food Detection

In an aging society, the number of caregivers in the nursing care industry is decreasing and the number of persons requiring nursing care is increasing, resulting in a serious labor shortage. In addition, measurement of the amount of food intake by caregivers is an important element of caregiving work, however, the measurement standard is not standardized and is inaccurate because it is done by caregivers' visual measurement. This research utilizes deep learning-based image recognition AI to automate the measurement of food intake. Specifically, a new framework that simultaneously performs the separate tasks of segmentation and object detection using ViT is proposed to perform segmentation of leftover food and detection of eating utensils. Furthermore, based on these recognition results, an algorithm to estimate the volume of the leftover food is proposed and its accuracy is evaluated. Experimental results show the effectiveness of the proposed Hybrid Segmentation method with ViT/8-small@224 at 89.25% mIoU in segmentation and 100.00% mAP, 95.47% mIoU for object detection. Future work includes updating the dataset and the leftover volume estimation method.