Multimodal learning for coffee bean classification using handcrafted features, image embeddings, and regional metadata

The growing global demand for single-origin coffee has intensified concerns over product adulteration, motivating the development of reliable and interpretable classification methods. This study proposes a multimodal learning framework for classifying Indonesian coffee beans by integrating handcrafted image features, deep image embeddings, and Geographical Indication (GI)–related regional metadata as contextual input features. Handcrafted descriptors capture shape, texture, and multi-scale phenotypic characteristics, while image embeddings provide high-level visual representations. Using a dataset of 9072 single-bean images spanning 54 coffee varieties, we evaluated multiple machine learning models across different modality combinations. Models trained solely on handcrafted features achieved an accuracy and F1-macro of 0.759 and 0.753, respectively. Incorporating regional metadata substantially improved performance (accuracy = 0.902, F1-macro = 0.900), while full multimodal fusion of handcrafted features, image embeddings, and regional metadata achieved the highest accuracy (0.991) and F1-macro (0.991). Across all model configurations, the inclusion of regional metadata consistently enhanced classification performance. While image-based models attained the highest predictive accuracy, the combination of handcrafted features and regional information offered improved interpretability. Feature selection using the Minimum Redundancy Maximum Relevance (MRMR) method and instance-level explanation using the Breakdown approach further highlighted the dominant influence of regional metadata. These findings demonstrate the effectiveness of multimodal learning for coffee bean classification and support the use of contextual regional information to balance accuracy and interpretability in agri-food authentication tasks.