FruitSight: A Millimeter Wave Radar Based Approach to Detect Occluded Fruits

The fruit picking robots of today rely on computer vision-based techniques to locate fruits. While these robots are fairly accurate in detecting fruits that their cameras can see, they miss a significant number of fruits that are either partially or fully occluded from the viewpoint of robot's cameras. To detect such occluded fruits, we propose to augment existing camera based approaches with a radar based approach because radio signals, unlike light, can propagate through objects, such as leaves and branches. We present FruitSight, a radar based approach that uses millimeter wave signals (60~64 GHz) to detect both visible and occluded fruits on trees. FruitSight works by transmitting a frequency modulated carrier wave (FMCW), receiving the reflection from the objects in front of it (such as fruit, leaves, branches etc.), and processing it to determine whether the reflection contained any signature of a fruit. The key intuition behind our approach is that as fruits have higher water content, they reflect a rather large amount of signal, and the strength of the reflected signal from a point just before and just after the fruit's physical location is quite low. FruitSight looks for variations in the reflected signal caused by such differences in the composition of fruits and leaves/branches to detect the presence of fruits. We collected a large dataset of almost 21K samples and present its extensive characterization to demonstrate the impact of fruits on reflected wireless signals. We have implemented FruitSight on real hardware and evaluated it extensively on real plants and fruits. Our results show that with just 20 transmissions of radio signal per detection round (which takes just 1 msec), FruitSight can achieve a true positive rate (TPR) of >97%.