EHAPZero: Ensemble Hierarchical Attribute Prompting-Based Zero-Shot Learning for Pest Recognition

Pest recognition is of great significance for achieving sustainable development in agriculture. Nevertheless, due to the wide variety of pest species, subtle interspecies differences, and significant intraspecies variations, existing artificial intelligence and Internet of Things (IoT) technologies can only recognize a small number of known pests effectively. In this article, we propose a zero-shot learning pest recognition framework based on ensemble hierarchical attribute prompting, termed EHAPZero. EHAPZero can identify pest images collected by IoT devices, and then transmit the recognition results to the IoT platform for terminal display. Specifically, the image recognition function is implemented by an attribute generation module (AGM), a hierarchical prompting module (HPM), and a semantic-visual interaction module (SVIM). AGM utilizes large language models to construct a knowledge graph of pests. It employs both node importance evaluation algorithms and manual methods to perform dual filtering on attribute nodes within the graph. Inspired by human knowledge reasoning, HPM dynamically predicts different hierarchical attributes of input images within the Transformer intermediate blocks. These predicted attributes are subsequently injected into the intermediate layer features of the Transformer as prompts. To achieve semantic disambiguation and knowledge transfer, SVIM employs a visual-guided semantic representation method and a semantic-guided visual representation method to strengthen cross-domain interaction between semantics and vision. Finally, the final prediction score is derived through ensemble of prediction results across different levels. Extensive experiments show that EHAPZero achieves the new state-of-the-art results on the real-word pest recognition benchmark. The codes are available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/jinqiwen/EHAPZero</uri>.