Video object segmentation and tracking of building structures in aerial motion imagery using deep learning

Tracking point and line features in image sequences is widely used in computer vision applications. But point and line features are difficult to detect and match in large urban scenes for geospatial localization and photogrammetry applications due to repeated structures like windows, roof tiles, textured facades (like walls), road and sidewalk linear structures. Instead of detecting and tracking handcrafted or learned local features we instead consider extraction and tracking of building or other manmade structures at the larger scale object level where there is more spatial context. Some of the challenges for accurate object detection and segmentation in aerial imagery include significant viewpoint changes, perspective warping, object occlusions and repetitive or similar objects like buildings and homes in a neighborhood. We test the performance of five recent deep learning models for video object segmentation including SAM2, DeAOT/DMAOT, RMem, STM and STCN for segmenting and tracking building structures in synthetic and real aerial motion imagery. We evaluate accuracy using both semantic masks and camera pose recovery using synthetic aerial imagery.