Towards Energy-Accuracy Scalable Multimodal Cognitive Systems

Transformer-powered multimodal artificial intelligence (MMAI) holds great promise for developing cognitive systems that can analyze and interpret data from various sensory modalities simultaneously. However, deploying MMAI on resource-constrained “edge” platforms poses significant challenges due to the intensive compute and memory requirements of transformer models, communication bandwidth limitations, real-time processing needs, and the intricacies of multimodal data fusion. To overcome these challenges, we introduce collaborative multimodal inference, leveraging the strengths of MMAI, edge computing, and cloud resources. Our solution introduces modality-aware accuracy-efficiency (AE) knobs, extending beyond multimodal sensors to individual subsystems within the edge system. We explore intersubsystem and intermodal interactions, investigating system-level AE tradeoffs in the presence of synergistic optimizations. Building on these insights, we present <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SysteMMX</monospace>, the first AE scalable cognitive system for efficient multimodal inference at the edge. In this letter, we present an in-depth case study centered around a multimodal system employing RGB and depth sensors for image segmentation. Our system, <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SysteMMX</monospace>, demonstrates significant energy savings—<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${1.8 \times }$ </tex-math></inline-formula> on the edge device and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${1.7\times }$ </tex-math></inline-formula> on the edge server—with an imperceptible application-level accuracy loss of less than 0.01%. Furthermore, <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SysteMMX</monospace> outperforms single-modality optimizations, achieving <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${1.2\times }$ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${1.8\times }$ </tex-math></inline-formula> more energy efficiency on the edge compared to RGB-only and Depth-only approaches, respectively, for similar levels of accuracy loss.