Multi-modal deep learning for paddy health assessment: fusing leaf imagery with tabular metadata using a factorized bilinear pooling approach

Global food security is largely based on the accurate and timely diagnosis of crop diseases, where paddy rice is an extremely essential staple of more than half of the world population. The conventional disease identification techniques tend to be laborious, time consuming and demand a great deal of domain knowledge, which becomes a bottleneck in the efficient management of the farms. Although deep learning [and especially Convolutional Neural Networks (CNNs)] have demonstrated a spectacular performance in automated classification of diseases based on leaf images, they tend to overlook important contextual features that are implicitly processed by agronomic experts. The visual defects of a disease might be unclear and this can greatly differ depending on factors like the genetic variety of the plant and the stage of development. We overcome this shortcoming by proposing a new multi-modal deep learning framework, Multi-Modal Factorized Bilinear Pooling (MFBP) model which is capable of a more holistic and precise paddy health measurement. The proposed method is the only one that combines high-level visual information obtained using leaf images and related tabular information, namely the paddy type and number of days. The MFBP model uses Factorized Bilinear Pooling (FBP) rather than the simple feature concatenation which commonly loses the complex relationship between different data types. This systematic method efficiently encodes all the complex interactions between all components of the visual and tabular features vectors in such a way that helps the model to pick up subtle, context-specific patterns. As an example, it will only be possible to educate the model that a specific visual blemish is predictive of a given disease through a specific species at a specific age. We test our model on the Paddy Doctor: Paddy Disease Classification dataset, which is a detailed public dataset comprising of more than 10,000 labeled images and containing relevant metadata, and thus it forms a perfect testing bed to conduct multi-modal research. Through our detailed experiments, we have shown that the proposed MFBP model is much better than a baseline model based on concatenation fusion, which proves that deep, multiplicative interactions can be best modeled in this task. The findings highlight the massive possibilities of multi-modes AI in the development of more robust, more accurate, and more context-aware diagnostic instruments and precision agriculture to enable more sustainable and productive agricultural activities.