Adaptive Feature Fusion and Enhancement Network for Food Nutrition Estimation

Nutrition estimation plays an important role in promoting food analysis and quality development. In recent years, with the development of deep learning and computer vision technologies, several vision-based nutrition evaluation methods have emerged. Compared with the traditional physical and chemical methods, these methods can quickly predict the content information of nutrients in food without damage. However, despite their promising evaluation performance, there are still some issues that need to be further conquered: first, how to effectively obtain comprehensive feature information about evaluated food, second, how to fully utilize different scale features and effectively fuse. To solve these problems, we propose an adaptive feature fusion and enhancement method, named ADFE. Specifically, the developed ADFE leverages the swin-transformer as the backbone network, which can obtain more effective features through window self-attention and shifted window self-attention. Besides, we design an adaptive fusion multimodal feature module, which can well handle the relationships between RGB and depth information. Furthermore, We propose a semantic-information enhancement module to enhance feature extraction and effectively fuse multiscale features. Finally, we conduct comprehensive experiments on the benchmark dataset Nutrition5k, compared with state-of-the-art methods, the mean value of the percentage mean absolute error (PMAE) for our method is 17.8<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula>. The PMAE of calories, carbs, and protein is 14.4<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula>, 21.0<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula>, and 19.4<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula>, marking improvements of 0.6<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula>, 1.4<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula>, and 1.6<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula>. Code and models are available online <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/123clown/ADFE</uri>.