ProFood: Progressive RGB-D fusion network for food detection in complex diet scenes

Food identification and detection are crucial for diet management and automated food analysis systems. Nevertheless, current methods face great challenges in dealing with the visual changes and frequent overlapping of food items in real scenes. To address these challenges, we introduce depth images into food detection for the first time, and propose a progressive multi-modal fusion detection method (ProFood). This method gradually integrates complementary information in food multimodal images to achieve accurate and robust detection in complex scenes. Specifically, ProFood first employs an attention mechanism to fuse and calibrate shallow features, thereby enhancing the perception of food texture and spatial information. A multimodal feature enhancement module is then designed to dynamically optimize the contributions of RGB, depth, and shallow fusion features, which improves deep semantic representation. Furthermore, a multiscale attention module is incorporated to enhance the model's adaptability to variations in food sizes. Compared with the baseline model, our method achieves 3.84 % improvement of mAP@50 on UECFOOD-256 data set and 1.65 % gain on ZSFOOD data set. In addition, our method is superior to the existing RGB-D method on the SUN RGB-D data set, showing superior robustness and generalization ability. • Depth image is first introduced into food detection in complex diet scene. • A progressive multimodal fusion framework is proposed. • SFCM module improves the perception of food information. • MFEM and MAM module improve the ability of multi-level semantic representation. • This method is superior to the baseline model on several public data sets.