Från bild- till videobaserad geolokalisering : Modellanpassning genom analys av flera bildrutor

This master’s thesis introduces a training-free framework for adapting image-based geo-localization models to video data through frame aggregation. To begin, a retrieval-based image geo-localization model, called vRuCLIP, is built and trained using a large-scale dataset of geo-tagged social media images from the site VK. VK is a Russian social media site. The training of vRuCLIP involves embedding images and GPS coordinates into a shared embedding space structured for image-to-GPS retrieval. GPS coordinates are embedded using a GPS encoder that utilizes random Fourier features to capture detailed patterns at different geographic scales. To embed images, a CLIP image encoder is used to create an image embedding vector, which is then projected into the GPS embedding space using a three-layer neural network. vRuCLIP is trained using contrastive learning, which is a process where related image– GPS pairs are pulled together while non-related pairs are pushed apart. Once vRuCLIP is trained, three methods for adapting it to video data are investigated. The first method involves constructing a single video representation by averaging the CLIP embeddings of frames sampled from the video. The second method averages the projected GPS embeddings of each frame. The third method entails computing an independent probability mass function over possible GPS coordinates for each frame, and combining these functions to get a final most likely prediction. To evaluate the proposed methods, a dataset of geo-tagged videos is created by linking each video to a geo-tagged image, posted near in time by the same user. A manually verified dataset of image–video pairs is also created, in order to assess the reliability of this geo-tagging method. The results demonstrate that frame aggregation is a viable and efficient method for extending image-based geo-localization models to video data, offering a solution suited to scenarios where computational resources or geo-tagged videos are limited.