Мультимодальний додаток здорового харчування та дієти на основі великої мовної моделі

Adhering to a healthy diet is one of the key factors towards improving lifespan and quality of life. However, we do not always know which meals are unhealthy. Existing food tracking applications require manual input of consumed dishes, which is time consuming and inaccurate. Neural networks have been shown to be good at estimating food ingredients and nutrition from a picture. In this work we propose a novel 2-stage approach that includes post-train adaptive neural network for image segmentation into ingredients, assisted by a large language model (LLM) that transforms ingredient information into nutritional value, gives insight into improving nutrition of the food consumed, and has chat interface, where user can clarify any questions. Both vision and language models are inferred fully on the user's device. Thus, no server connection is required, and the user data remains private. We develop several prompts for nutritional value extraction, and analyze Qwen 2.5 0.5, 1.5, 3 billion parameter LLMs. To score nutritional value estimates we propose LLM-judge method. The final prompt improves nutritional value estimation score from 2.66 to 3.68, when using Qwen 2.5 3 billion model. The proposed application is more user friendly and provides nutrition tracking with less effort, when compared to existing applications, enabling faster food tracking with detailed information on improving nutrition.