SMURF: Federated Multimodal Retrieval for Scientific Data via Embedding Alignment

Scientific data spans heterogeneous modalities and is distributed across diverse storage systems. Vector databases have emerged as powerful tools to index high-dimensional embeddings for semantic search, but most existing systems are centralized and focus on indexing a single modality. We introduce SMURF, a system for federated multimodal indexing and retrieval. SMURF aligns pre-computed embeddings from different scientific modalities into a unified embedding space. Our method extends semi-supervised learning techniques to connect disparate embeddings by combining geometry-preserving alignment with weak supervision using a small set of semantically matched data. We evaluate SMURF on cross-modal retrieval tasks using scientific datasets and find that aligned embeddings substantially improve performance—on average, top-5 accuracy nearly doubles (19.18%→38.82%), and nDCG@10, a ranking quality metric, increases by 78% (0.18→0.32) across various retrieval settings. These results highlight the promise of bridging modality gaps to support more effective scientific discovery in federated environments.