Multisensory machine intelligence

Abstract The future of artificial intelligence demands a paradigm shift toward multisensory perception—to systems that can digest ongoing multisensory observations, that can discover structure in unlabeled raw sensory data, and that can intelligently fuse useful information from different sensory modalities for decision‐making. While we humans naturally perceive the world by looking, listening, touching, smelling, and tasting, traditional forms of machine intelligence mostly focus on a single sensory modality, particularly vision. Therefore, my research, which I refer to as multisensory machine intelligence, seeks to bridge this gap by empowering machines to emulate and enhance human capabilities in seeing, hearing, and feeling, ultimately enabling them to comprehensively perceive, understand, and interact with multisensory world.