Quantitative Analysis of the Labeling Quality of Biological Images for Semantic Segmentation Based on Attribute Agreement Analysis

Semantic segmentation in biological images is increasingly common, particularly in smart agriculture, where deep learning model precision is tied to image labeling quality. However, research has largely focused on improving models rather than analyzing image labeling quality. We proposed a method for quantitatively assessing labeling quality in semantically segmented biological images using attribute agreement analysis. This method evaluates labeling variation, including internal, external, and overall labeling quality, and labeling bias between labeling results and standards through case studies of tomato stem and group-reared pig images, which vary in labeling complexity. The process involves the following three steps: confusion matrix calculation, Kappa value determination, and labeling quality assessment. Initially, two labeling workers were randomly selected to label ten images from each category twice, according to the requirements of the attribute agreement analysis method. Confusion matrices for each image’s dual labeling results were calculated, followed by Kappa value computation. Finally, labeling quality was evaluated by comparing Kappa values against quality criteria. We also introduced a contour ring method to enhance Kappa value differentiation in imbalanced sample scenarios. Three types of representative images were used to test the performance of the proposed method. The results show that attribute agreement analysis effectively quantifies image labeling quality, and the contour ring method improves Kappa value differentiation. The attribute agreement analysis method allows for quantitative analysis of labeling quality based on image labeling difficulty, and Kappa values can also be used as a metric of image labeling difficulty. Dynamic analysis of image labeling variations over time needs further research.