InsectRoleVision: A computer vision system to study arthropod diversity and functional roles

Arthropods make up the majority of species on Earth. To study their diversity and ecological roles in ecosystems, bulk sampling is commonly used to collect large numbers of specimens. Processing these samples is labor-intensive and time-consuming, often delaying timely decision-making in ecosystem monitoring. Automated detection systems offer a promising alternative to support sample processing; however, most existing systems still have two major limitations. First, the pipelines for localizing and classifying arthropods in images, whether single- or double-stage, are limited. Second, the prediction results often lack information about functional roles. Therefore, we developed InsectRoleVision , a more expert-centered and reliable system that enables inference of arthropod diversity based on their functional roles. To develop the system, an image dataset with taxonomic resolution was designed and created to support conclusions about the functional roles of the animals. Both single-stage and double-stage recognition pipelines were compared. For single-stage detection and the first stage of double-stage detection, four YOLO models and a transformer were evaluated to localize and classify the arthropods in each image. In double-stage detection, the region of interest (RoI) was cropped into individual images after localization and used to compare four classification models: InceptionV3, ResNet, MobileNet, and VGG19. A logic block pipeline was connected to the prediction results to further infer the richness and proportionality of each class or taxon with respect to their functional roles. YOLOv11 was the best-performing model, achieving over 93% mAP, precision, and recall in localizing arthropods in the images. InceptionV3 was the best-performing classifier, achieving 80% precision and recall in classifying more than 43,000 cropped images of arthropods. There was no significant difference between the results of single- and modular double-stage detection strategies. Therefore, the choice between strategies depends on the intended application: single-stage detection provides real-time results and is suitable for real-time detection applications, while double-stage detection allows a human expert to review the detection proposal and refine the classification result. InsectRoleVision has adopted the YOLOv11-InceptionV3 architecture, which is more flexible and human-centered, allowing quick access to both arthropod diversity and ecological roles. • Annotating the datasets based on the ecological and economic roles of arthropods increases the relevance of model predictions for applied use cases. • The YOLO models outperform the Transformer-based architecture. • YOLOv11 achieves excellent results in both localization (double-stage detection) and recognition (single-stage detection) of arthropods from scanned images. • InsectRoleVision uses a modular double-stage detection strategy with the YOLOv11-InceptionV3 architecture and demonstrates significantly better performance. • The InsectRoleVision system does not differ significantly from manual studies of two urban ecosystems in determining the diversity and functional roles of arthropods