Data Extraction for Vision-Based Learning from Demonstration

Automation in assembly production is highly desirable but limited. Using man- ual labor puts workers into risk of injury when performing monotonous tasks. Installation and/or reprogramming robots is both expensive and yield uncertain profit. Hence, using a learning from demonstration (LfD) solution with collabora- tive robots (cobots) may increase productivity and safety. However, LfD often re- lies on large volumes of high-dimensional data, which significantly increases the computational load when training deep learning models. Therefore, this thesis investigates custom built methods for data extraction for visual-based learning from demonstration. We present two novel machine learning based pipelines that are designed to extract start and end position of object movements from video demonstrations. One pipeline is based on the Segment Anything Model 2 (SAM2) and the other on the Hand-Object Detector (HOD) trained on the 100 Days of Hands (100DOH) dataset. The thesis explores the following key questions: Can the pipelines gen- eralize to cluttered scenes, gloved hands or indecisive hand movements? How do they compare in performance? Can the performance of the HOD-based pipeline be improved by training on custom data or a combination of 100DOH and cus- tom data? What visual cues are most important for the data extraction pipelines? The SAM2-based pipeline uses detection, video propagation and depth filter- ing to track object movements, while the HOD-based pipeline focuses on hands and their interaction with objects using contact state detection. Both pipelines were evaluated on a evaluation dataset that we developed, containing different hand appearances, object types and movement complexities. We conclude that the SAM2-based pipeline generally outperforms the HOD- based one, mainly due to its use of temporal information but also because its per- formance was not influenced by the appearance of the hand. The HOD-based pipeline did improve dramatically when trained with an additional, targeted dataset, which shows how important it is to have domain-specific data. As fu- ture work we suggest combining the strengths of the two pipelines, hand-object contact to lead the focus and object segmentation for increased robustness.