Grounded Language Generation from Raw Sensory Input in Embodied AI Systems
Language produced by a system that has never acted in the world describes that world from the outside. Embodied systems that perceive through their own sensors and change what they perceive by acting offer a different setting for language generation, one in which the meaning of a term can be tied to sensory evidence and to the actions the term licenses. This article examines what such grounding requires and how far current systems supply it. We distinguish three requirements, namely perceptual anchoring of reference, action conditioned meaning and situated deixis, and show that pipelines which caption an image and pass the caption to a planner satisfy only the first. End-to-end architectures that map sensory observations and instructions directly to actions and to language remove the intermediate text bottleneck, and we compare eight such systems by input, output and training signal. We then examine the data available, which is smaller than the corpora used for text pre-training by five orders of magnitude, and the evaluation problem, where reference based caption metrics reward fluency and cannot detect a description of an object that is not present. A taxonomy of grounding failures is proposed together with the diagnostic each one requires, and seven open problems are stated.
Authors
- Manasy Jayasurya
Publication Details
- Journal
- Eduschool Journal of Artificial Intelligence Research (EJAIR)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.63090/ejair/3139.7336.0011
- Primary Topic
- Multimodal Machine Learning Applications
- Type
- article
- Field-Weighted Citation Impact
- 0.00