AeroTea-VLM: a UAV-assisted multimodal visual language model for intelligent disease and pest management in terraced tea plantations

Abstract Current vision-language models (VLMs) that are used for UAV agricultural inspection encounter significant challenges. These problems come from scale distortion when drones fly at different heights, and also from really long delays that happen during visual action planning. In this paper, we introduce AeroTea-VLM, which fixes these issues by using a special new idea called "heavy perception, light control." To make sure perception stays the same regardless of scale in 3D terraced farm areas, we came up with two new methods: Altitude-Aware Position Encoding (Alt-PE) and Cross-Altitude Topological Attention (CATA). These two methods work together by adding real-time altimeter and depth data, which helps cut down on feature aliasing problems. For how the drone moves around, our Embodied Chain-of-Action (CoA) module gets rid of the slow autoregressive image generation step. Instead, it just outputs text commands directly. This change cuts total end-to-end delay by 86.5% (down to 385 ms), and it also keeps peak GPU memory usage at 14.2 GB. We tested this whole system on a new dataset that we collected ourselves, which we call AeroTea-Multimodal RGB-D. The tests show that AeroTea-VLM gets 88.5% accuracy when identifying plant diseases, and has an 84.2% success rate on reasoning tasks. Both of these numbers are better than what existing baseline methods can get.