OmniQuery: Enabling Question Answering on Personal Memory by Augmenting Multimodal Album Data

We present OmniQuery, an interactive system that augments users’ personal photo albums and enables free-form question answering on users’ past memories. OmniQuery processes multimodal media data in personal albums, aggregates them into related episodic memory databases in different levels, and infers semantic knowledge including personal facts like social relationships, preferences, and experiences. OmniQuery then allows users to interact with their database using natural language, giving media that directly matches the query or an exact answer supported by related media as a result.