NETSI: Nonequidistant Time Series Imaging-Based Multisensor Fusion Method for Smart Fruit Sorting

Considering the various fruit features and the complex sorting environments, low-cost and efficient multiclass fruit sorting remains a challenging topic. This article proposes a multisensor fusion method to achieve accurate and stable classification of multiclass fruits in interference environments, by using fusion and classification based on an innovative nonequidistant time series imaging (NETSI) algorithm. First, we construct a two-channel imaging scheme that includes inner product-based NETSI (INETSI) and outer product-based NETSI (ONETSI), which guarantees the robustness and generalization of the fruit sorting signals in interference environments and enhances the anti-interference ability of fruit sorting. Then, a NETSI-based multisensor fusion framework is proposed aiming to further improve the efficiency of fruit sorting with multiclass fruits in interference environments. The framework fuses INETSI and ONETSI images with visual images and feeds into a deep neural network for accurate recognition. Finally, based on the proposed framework, we build a nondestructive, efficient, and low-cost fruit sorting device that is applied to facilitate a fruit picking robot. The experiments show that the proposed method outperforms existing methods and enables efficient and reliable sorting of multiclass fruits in interference environments.