Opportunities to design better computer vison-assisted food diaries to support individuals and experts in dietary assessment: An observation and interview study with nutrition experts

Automatic visual recognition for photo-based food diaries is increasingly prevalent. However, existing tools in food recognition often focus on food classification and calorie counting, which may not be sufficient to support the variety of food and healthy eating goals people have. To understand how to better design computer-vision-based food diaries to support healthy eating, we began to examine how nutrition experts, such as dietitians, use the visual features of food photos to evaluate diet quality. We conducted an observation and interview study with 18 dietitians, during which we asked the dietitians to review a seven-day photo-based food diary and fill out an evaluation form about their observations, recommendations, and questions. We then conducted follow-up interviews to understand their strategies, needs, and challenges of photo diary review. Our findings show that dietitians used the photo features to understand long-term eating patterns, diet variety, eating contexts, and food portions. Dietitians also adopted various strategies to achieve these understandings, such as grouping photos to find patterns, using color to estimate food variety, and identifying background objects to infer eating contexts. These findings suggest design opportunities for future compute-vision-based food diaries to account for dietary patterns over time, incorporate contextual information in dietary analysis, and support collaborations between nutrition experts, clients, and computer vision systems in dietary review and provide individualized recommendations.