Automated Food Nutrition Retrieval for Healthcare and Medical Applications using Image-Based Weight Prediction Models

Malaysia’s diverse and rich cuisine is a fundamental part of Malaysia’s culture. However, there are many Malaysia food that are high in calories and sugar contents, which can lead to health issues such as diabetes and obesity. Not only that, traditional calorie estimation methods such as manual food tracking are ineffective due to lack of a complete Malaysia food dataset and portion size estimation errors. To address the difficulties mentioned above, this study aims to propose a machine learning-based solution for automated food nutrition retrieval using image-based weight prediction models. First, data collection for Malaysia food images is performed across five food classes. Next, YOLOv11, which is an instance segmentation model is trained for food identification task and three gradient boosting algorithms, XGBoost, CatBoost and LightGBM are trained to identify best model for food weight prediction. Both food recognition model and weight prediction model are evaluated based on performance metrics such as precision and recall. Based on results obtained, food recognition model has achieved 99.20% precision and 99.50% recall while XGBoost has been chosen as best model to estimate food weight with MAE of 6.21g, RMSE of 8.00g and MAPE of 4.62%. After the food type and food weight are identified, they are provided as inputs to an application programming interface (API) that is created based on Malaysia food composition database to obtain nutritional information such as calories, fat and protein in healthcare applications.