MIRACLE: An Online, Explainable Multimodal Interactive Concept Learning System
We present MIRACLE, a system for online, interpretable visual concept and video action recognition. Through a chat interface, users query the recognition system with an uploaded image or video. For images, MIRACLE returns concept predictions from its structured knowledge base, justifying its predictions with heatmaps and natural language-based attribute detections. For videos, MIRACLE predicts an action and justifies its prediction with time varying entity-entity relations. With its ability to learn new concepts in an online, few-shot manner and its support of dynamic changes to its knowledge base, MIRACLE represents a step forward in interpretable multimodal learning systems.
Authors
- Derek Hoiem (ORCID: https://orcid.org/0000-0001-6260-5708)
- Michal Shlapentokh-Rothman
- Nanyun Peng (ORCID: https://orcid.org/0000-0002-8509-6595)
- M. Sidhu (ORCID: https://orcid.org/0009-0002-6021-4134)
- Elias Stengel-Eskin (ORCID: https://orcid.org/0000-0002-6689-505X)
- Zhenhailong Wang (ORCID: https://orcid.org/0000-0002-4704-5455)
- Kuan-Hao Huang (ORCID: https://orcid.org/0000-0002-1634-9237)
- Lydia B. Chilton (ORCID: https://orcid.org/0000-0002-1737-1276)
- Vivian Liu (ORCID: https://orcid.org/0000-0001-5328-0120)
- Carl Vondrick (ORCID: https://orcid.org/0000-0003-1139-9208)
- Abhay Zala
- Xuanming Zhang (ORCID: https://orcid.org/0000-0003-2776-0344)
- Utkarsh Mall (ORCID: https://orcid.org/0009-0002-5302-0657)
- Camille Cobb (ORCID: https://orcid.org/0000-0001-5349-8250)
- Heng Ji (ORCID: https://orcid.org/0000-0002-0464-7966)
- Karrie G. Karahalios (ORCID: https://orcid.org/0000-0001-8788-3405)
- Kai-Wei Chang (ORCID: https://orcid.org/0000-0001-5365-0072)
- Jiateng Liu (ORCID: https://orcid.org/0000-0002-7946-1381)
- Yangyi Chen (ORCID: https://orcid.org/0000-0001-8607-1886)
- Ansel Blume
- Yin Da (ORCID: https://orcid.org/0000-0001-7215-6563)
- Yao Xiao (ORCID: https://orcid.org/0000-0002-8258-0736)
- Zhen Zhu (ORCID: https://orcid.org/0000-0003-1557-8473)
- Zhou Yu (ORCID: https://orcid.org/0000-0002-1524-5890)
- Te-Lin Wu
- R. P. Sinha
- Xiaomeng Jin
- Khanh Duy Nguyen (ORCID: https://orcid.org/0000-0002-7427-5272)
- Jeonghwan Kim (ORCID: https://orcid.org/0009-0002-7277-2168)
- Mohit Bansal
Institutions
- University of North Carolina at Chapel Hill (US)
- University of North Carolina Health Care (US)
- Urbana University (US)
- UCLA Health (US)
- Columbia University (US)
Publication Details
- Published
- 2024-10-26
- DOI
- https://doi.org/10.1145/3664647.3684993
- Citations
- 1
- Primary Topic
- Multimodal Machine Learning Applications
- Type
- article
- Field-Weighted Citation Impact
- 0.25