Multi-modal multi-task federated foundation models for next-generation extended reality systems: towards privacy-preserving distributed intelligence in AR/VR/MR

Abstract Extended reality (XR) systems, encompassing virtual reality (VR), augmented reality (AR), and mixed reality (MR), offer a transformative interface for immersive, multi-modal, and embodied human-computer interaction. In this paper, we envision that multi-modal multi-task (M3T) federated foundation models (FedFMs) can offer transformative capabilities for XR systems through integrating the representational strength of M3T foundation models (FMs) with the privacy-preserving and personalized model training principles of federated learning (FL). To this end, we first present the modular architecture of FedFMs, which entails different coordination paradigms for model training and aggregations. Afterward, we codify XR challenges that affect the implementation of FedFMs under the SHIFT dimensions: (1) $$\underline{{\bf{S}}}{\rm{ensor}}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:munder> <mml:mrow> <mml:mi>S</mml:mi> </mml:mrow> <mml:mrow> <mml:mo>̲</mml:mo> </mml:mrow> </mml:munder> <mml:mi>ensor</mml:mi> </mml:mrow> </mml:math> and modality diversity, (2) $$\underline{{\bf{H}}}{\rm{ardware}}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:munder> <mml:mrow> <mml:mi>H</mml:mi> </mml:mrow> <mml:mrow> <mml:mo>̲</mml:mo> </mml:mrow> </mml:munder> <mml:mi>ardware</mml:mi> </mml:mrow> </mml:math> heterogeneity and system-level constraints, (3) $$\underline{{\bf{I}}}{\rm{nteractivity}}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:munder> <mml:mrow> <mml:mi>I</mml:mi> </mml:mrow> <mml:mrow> <mml:mo>̲</mml:mo> </mml:mrow> </mml:munder> <mml:mi>nteractivity</mml:mi> </mml:mrow> </mml:math> and embodied personalization, (4) $$\underline{{\bf{F}}}{\rm{unctional}}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:munder> <mml:mrow> <mml:mi>F</mml:mi> </mml:mrow> <mml:mrow> <mml:mo>̲</mml:mo> </mml:mrow> </mml:munder> <mml:mi>unctional</mml:mi> </mml:mrow> </mml:math> /task variability, and (5) $$\underline{{\bf{T}}}{\rm{emporality}}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:munder> <mml:mrow> <mml:mi>T</mml:mi> </mml:mrow> <mml:mrow> <mml:mo>̲</mml:mo> </mml:mrow> </mml:munder> <mml:mi>emporality</mml:mi> </mml:mrow> </mml:math> and environmental variability. We then illustrate the manifestation of these dimensions across a set of emerging and anticipated applications of XR systems. Finally, we propose evaluation metrics, dataset requirements, and design tradeoffs necessary for the development of resource-efficient FedFMs in XR ecosystems.