NoisyBox: a dataset for analyzing structured spatial annotation noise in bounding boxes

In aviation and other safety-critical domains, computer vision systems must operate with high reliability, but are often trained on noisy and inconsistent bounding box annotations. NoisyBox is introduced as an open dataset for evaluating classification robustness under noisy bounding box annotations in fine-grained, safety-critical visual domains. The dataset consists of cockpit instrument images collected from flight simulation software and labeled by eight annotators, each contributing a single annotation per image. Due to lack of redundancy and expert oversight, the dataset contains significant bounding box noise. A separate expert-labeled dataset serves as the gold standard for benchmarking purposes. Evaluation of standard classification models revealed substantial sensitivity to annotation quality. Models trained on gold-standard data showed sharp performance degradation when tested on NoisyBox, with average F1 scores dropping from 0.98 to 0.38 (VGG16) and from 0.99 to 0.48 (ResNet50). Traditional ensemble methods such as majority voting were rendered ineffective by structured spatial noise (F1: 0.42), as geometric distortions led to correlated errors across models, failing to surpass the most robust single backbone (ResNet50, F1: 0.48). In contrast, a class-specific expert model selection approach provided a genuine performance gain increasing the average F1 score to 0.52 which demonstrates the benefit of backbone specialization for different classes. To address structured spatial noise, a classifier-agnostic low-supervision coverage-quality analysis framework is evaluated to identify bounding box types—full, partial, none, and multi-object. By leveraging intermediate feature representations rather than raw logits, the approach achieved a 0.92 macro-averaged F1 with 80% labeled data and maintained an F1 score of 0.83 with just 5% supervision. However, the 1% labeled setting should be interpreted cautiously because minority coverage categories become extremely sparse under this condition. NoisyBox provides a realistic and challenging benchmark for studying the impact of annotation quality and supports the development of robust learning strategies in domains where repeated labeling or expert annotation is impractical.