MechLMM: A Collaborative Knowledge Framework for Enhanced Data Fusion in Multi-Robot Systems Using Large Multimodal Models

Multi-robot systems have become essential for dynamic tasks, especially robot-assisted disaster response missions, yet they often face challenges in achieving effective collaboration, inconsistent object detection, difficulty retaining comprehensive contextual data, and inefficient data fusion from multiple robots while a human operators take a key role for critical high level decisions. These systems also struggle with real-time scenarios where critical information is lost or oversimplified during data processing. This paper presents MechLMM (Mech Large Multimodal Model), a framework designed to address these challenges by enabling seamless collaboration between robots while maintaining a unified view of the environment. MechLMM leverages a Large Multimodal Model (LMM) to process raw sensor data, such as video inputs, into structured, human-understandable formats enriched with timestamps and location data. The framework integrates a shared knowledge database, allowing operators to retrieve detailed historical data through dynamic queries. A Multi-Robot Interaction Module facilitates communication between robots, merging and refining data inputs for cohesive object tracking, while a 3D Shared Information Map enhances situational awareness for human operators. Experimental results demonstrate that MechLMM improves object detection accuracy, data consistency, and real-time decision-making in multi-robot systems.