D2.2 Retrieval model for conversational style queries

This report presents research and experimental developments under WP2, Task 2, focusing on factual information retrieval (FIR) with conversational queries. We address open challenges including enhancing large language model (LLM) efficiency, mitigating hallucinations, identifying bias, and developing methodologies for semi-automatic creation of FIR training and benchmark datasets. Building upon prior work in T2.1, we extend a speech-to-text processing pipeline that enables reuse of existing text-based FIR systems for conversational speech data, using a BGE-M3 multilingual retrieval model. Two alignment strategies between speech and text embeddings are explored, with input-space alignment yielding superior results. Additionally, we propose a methodology for constructing a multimodal QA dataset from natural podcast conversations, advancing resources for FIR on spontaneous speech. Our work also expands on synthetic dataset creation using Retrieval-Augmented Generation (RAG) techniques, contributing to better contextualization and trustworthiness of LLM-based systems. Further, we investigate document plausibility and the associated findability bias, providing strategies to improve retrieval robustness and diversity. Finally, we explore two emerging research directions: introducing span-copying mechanisms to enhance LLM efficiency and reduce hallucinations and designing fine-grained retrieval models combining LLM supervision with lightweight, interpretable retrieval architectures. These efforts collectively advance multimodal, reliable, and efficient FIR systems suitable for real-world conversational applications.