Babel: A Scalable Pre-trained Model for Multi-Modal Sensing via Expandable Modality Alignment
This paper presents Babel, the expandable modality alignment model, specially designed for multi-modal sensing. While there has been considerable work on multi-modality alignment, they all struggle to effectively incorporate multiple sensing modalities due to the data scarcity constraints. How to utilize multi-modal data with partial pairings in sensing remains an unresolved challenge.
Authors
- Suman Banerjee (ORCID: https://orcid.org/0000-0002-5548-8862)
- Lili Qiu
- Yifan Yang (ORCID: https://orcid.org/0000-0002-5481-2851)
- Mo Li (ORCID: https://orcid.org/0000-0002-6047-9709)
- Ting Cao (ORCID: https://orcid.org/0000-0002-9107-013X)
- Shiqi Jiang (ORCID: https://orcid.org/0000-0002-4685-9633)
- Shenghong Dai (ORCID: https://orcid.org/0000-0001-6316-3980)
Institutions
- University of Wisconsin–Madison (US)
- Hong Kong University of Science and Technology (HK)
- Microsoft Research Asia (China) (CN)
- The University of Texas at Austin (US)
Publication Details
- Published
- 2025-05-04
- DOI
- https://doi.org/10.1145/3715014.3722068
- Citations
- 2
- Primary Topic
- Human Pose and Action Recognition
- Type
- article
- Field-Weighted Citation Impact
- 2.55
Funders
- U.S. Department of Commerce