T-EAGLE: Capturing Temporal Narratives via Sequence Captioning and Text Matching

There is a growing need to retrieve specific events or information from personal lifelog data, but this is particularly challenging due to the massive scale and the passive nature of data capture by lifelogging devices. Current systems typically rely on image similarity for single, isolated images, which struggle to capture the user intent expressed in natural language and the semantic links between the images and activities occurring over time. To address this issue, we propose a novel lifelog retrieval framework that explicitly combines both visual and temporal similarity in a multi-stage process, shifting the focus from single images to coherent sequences of actions. Our approach uses image embeddings to initialize a set of candidate images. Importantly, the system then re-evaluates the query similarity based on action descriptions which contain temporal information across image sequences. Action captioning, integrated into the indexing process, captures richer temporal and semantic context, allowing the system to distinguish between visually similar but semantically distinct events. Additionally, the system incorporates an evidence-based question answering mechanism, in which the narratives of the retrieved sequences provide contextual grounding for the answering model. The paper proposes a hybrid retrieval framework that combines image similarity for candidate initialization and visual-textual similarity for event retrieval. The integration of action descriptions enables language-based temporal representation of events. These are extracted offline through semantic content analysis and serve as the basis for building an evidence-based Question Answering module using these narratives as context. This approach helps bridge the gap between user intent and the multimodal, temporally structured nature of lifelog data.