VILOD: A Visual Interactive Labeling Tool for Object Detection

The advancement of Object Detection (OD) using Deep Learning (DL) is often hindered by the significant challenge of acquiring large, accurately labeled datasets, a process that is time-consuming and expensive. While techniques like Active Learning (AL) can reduce annotation effort by intelligently querying informative samples, they often lack transparency, limit the strategic insight of human experts, and may overlook informative samples not aligned with an employed query strategy. To mitigate these issues, Human-in-the-Loop (HITL) approaches integrating human intelligence and intuition throughout the machine learning life-cycle have gained traction. Leveraging Visual Analytics (VA), effective interfaces can be created to facilitate this human-AI collaboration. This thesis explores the intersection of these fields by developing and investigating VILOD: A Visual Interactive Labeling tool for Object Detection, a system designed to support expert users in the OD annotation process by integrating interactive visualizations, with AL suggestions and guidance. VILOD utilizes components such as a t-SNE projection of image features, together with uncertainty heatmaps and model state views. Enabling users to explore data, interpret model states, AL suggestions, and implement diverse sample selection strategies within an iterative HITL workflow for OD. An empirical investigation using comparative use cases demonstrated how VILOD, through its interactive visualizations, facilitates the implementation of distinct labeling strategies by making the model’s state and dataset characteristics more interpretable (RQ1). Furthermore, the study showed that different visually-guided labeling strategies employed within VILOD result in competitive OD performance trajectories compared to an automated uncertainty sampling AL baseline (RQ2). Specifically, the Balanced Guidance Integration strategy, which synthesized information from all available cues, achieved the highest final model performance. Suggesting that human guidance effectively supported by interactive VA can offer an edge in performance and provide important benefits such as quality control and strategic depth in the OD annotation process. This work contributes a novel tool and empirical insight into making the HITL-AL workflow for OD annotation more transparent, manageable, and potentially more effective.