Joint plant-spraypoint detector with ConvNeXt modules and HistMatch normalization

Abstract Context Serrated tussock ( Nassella trichotoma ) is a weed of national significance in Australia which offers little to no nutritional value to livestock, and has the potential to reduce carrying capacity and agricultural return of infested pastures. Aims The aim of this study was to adapt existing Convolutional Neural Networks (CNNs) for plant segmentation and spraypoint detection in the challenging environments of pastures. Methods CNNs that were designed for joint plant and stem segmentation in crop fields were repurposed for dual-task applications in pastures. Given the poor performance of these models in complex pasture environments, a new model drawing inspiration from the recently proposed ConvNeXt was developed, tested for its effectiveness on unseen field data, and enhanced with a novel normalization technique, called HistMatch. Key results Experimentation demonstrated that unlike pre-existing models, which were designed for the simpler environments encountered in early-stage crop fields, our model was able to generalize well to growing conditions not seen during training, achieving 0.807 mIoU and 0.796 F1-score for the plant and spraypoint tasks respectively. This is in comparison to pre-existing models, which achieved 0.270 - 0.454 mIoU and 0.073 - 0.496 F1-score for the same tasks. These results were further improved to 0.854 mIoU and 0.806 F1-score using HistMatch normalization. In spite of greater model complexity, our model had a inference time of 15.7 ms which was comparable to pre-existing models, and suitable for real-time applications. Conclusion Models with greater complexity are required for the relatively complex environments encountered in pastures, but this greater complexity need not come at the expense of real time capability. HistMatch normalization can improve model accuracy, and is particularly effective in cases where models are struggling to generalize well to testing conditions that vary significantly from those seen during training. Implications and impacts The successful adaptation and improvement of CNNs for weed management in pastures could significantly reduce the reliance on blanket herbicide application. HistMatch normalization could also be considered for other agricultural applications, including weed management and disease detection in crop fields and orchards.