D4: Text-guided diffusion model-based domain adaptive data augmentation for vineyard shoot detection

In agricultural practices, plant phenotyping using object detection models is gaining attention, plant phenotyping is a technology that accurately measures the quality and condition of cultivated crops from images, contributing to the improvement of crop yield and quality, as well as reducing environmental impact. However, collecting the training data necessary to create generic and high-precision models is extremely challenging due difficulties associated with annotations and the diversity of domains. Such difficulties arise from the unique shapes and backgrounds of plants, as well as the significant changes in appearance due to environmental conditions and growth stages. Furthermore, it is difficult to transfer training data across different crops, and although machine learning models effective for specific environments, conditions, and crops have been developed, they cannot be widely applied in real-world conditions. Therefore, in this study, we propose a generative artificial intelligence data augmentation method (D4) and investigated its application towards a shoot detection task in a vineyard. D4 uses a pre-trained text-guided diffusion model based on a large number of original images culled from video data collected by unmanned ground vehicles or other means, and a small number of annotated datasets. The proposed method generates new annotated images with background information adapted to the target domain while retaining annotation information necessary for object detection. In addition, D4 overcomes the lack of training data in agriculture, including the difficulty of annotation and diversity of domains. We confirmed that this generative data augmentation method improved the mean average precision by up to 28.65% for the BBox detection task and the average precision by up to 13.73% for the keypoint detection task for vineyard shoot detection. D4 generative data augmentation is expected to simultaneously solve the cost and domain diversity issues of training data generation for agricultural applications and improve the generalization performance of detection models. • Proposed novel data augmentation method D4 using text-guided diffusion model. • Analyzed detection accuracy using D4 for BBox detection and keypoint detection. • D4 improved detection accuracy and demonstrated effectiveness of domain adaptation. • D4 uses automatic selection with DreamSim to maintain generated image quality. • D4 overcomes lack of training data in agriculture.