A Provenance-Aware Framework for Cross-Modal Knowledge Discovery from Text, Images, and Audio-Visual Dialogue

A large share of institutional and educational knowledge is not stored as clean digital text but is embedded in scanned documents, photographed whiteboards, signage, and recorded human dialogue such as lectures, interviews, and meetings. Classical Knowledge Discovery in Databases (KDD) and text-mining pipelines do not extend directly to this material, and independent single-modality pipelines cannot recognize that a fact appearing in both an image and a spoken utterance is one fact rather than two. This paper presents a provenance-aware framework for knowledge discovery from text, images, and audio-visual dialogue. The framework extracts candidate entity and relation mentions from each modality using established open-source components (scene-text detection, optical character recognition, object detection, automatic speech recognition, speaker diarization, and transformer-based language understanding), represents them under a common mention schema, and aligns them across modalities through a weighted combination of temporal, semantic, and contextual evidence. A confidence-aware fusion step merges corroborating mentions, explicitly flags conflicting evidence instead of silently overwriting it, and persists each reconciled fact in a knowledge graph whose edges carry their source modality, source segment, and fused confidence as queryable properties. A hybrid retrieval workflow combining dense vector search with graph traversal supports semantic search, summarization, and question answering with traceable evidence. We describe the architecture, the data contracts, the alignment and fusion procedures, and a component- and system-level evaluation protocol for the prototype.