Enhancing Lost and Found Systems with Multi-Modal Deep Learning: Integrating SBERT and Siamese Networks for Improved Semantic Matching

Returning lost and found items in public spaces is challenging with traditional methods, and while technological advancements have led to systematic approaches, they often rely on query-based searches or image classification. This research provides a solution that combines textual and visual data to improve the semantic matching of lost and found items to address these problems. Three deep learning models for image similarity, text similarity, and fusion are implemented in a progressive web application (PWA) to support user data input and matching alerts. A fusion model was created by combining the SBERT model, which was refined using a dataset of 2,600 lost and found description pairs both for English and Sinhala languages, and the Siamese network, which was trained on 848 bag images using MobileNetV2. This fusion model also incorporates location and time features to give priority to recent activities and places to enhance matching accuracy. A neural network