Grounded Language Generation from Raw Sensory Input in Embodied AI Systems

Language produced by a system that has never acted in the world describes that world from the outside. Embodied systems that perceive through their own sensors and change what they perceive by acting offer a different setting for language generation, one in which the meaning of a term can be tied to sensory evidence and to the actions the term licenses. This article examines what such grounding requires and how far current systems supply it. We distinguish three requirements, namely perceptual anchoring of reference, action conditioned meaning and situated deixis, and show that pipelines which caption an image and pass the caption to a planner satisfy only the first. End-to-end architectures that map sensory observations and instructions directly to actions and to language remove the intermediate text bottleneck, and we compare eight such systems by input, output and training signal. We then examine the data available, which is smaller than the corpora used for text pre-training by five orders of magnitude, and the evaluation problem, where reference based caption metrics reward fluency and cannot detect a description of an object that is not present. A taxonomy of grounding failures is proposed together with the diagnostic each one requires, and seven open problems are stated.

Authors

Publication Details

Journal
Eduschool Journal of Artificial Intelligence Research (EJAIR)
Published
2026-11-18
DOI
https://doi.org/10.63090/ejair/3139.7336.0011
Primary Topic
Multimodal Machine Learning Applications
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Grounded Language Generation from Raw Sensory Input in Embodied AI Systems

Manasy Jayasurya
Eduschool Journal of Artificial Intelligence Research (EJAIR)
Multimodal Machine Learning Applications
article

Grounded Language Generation from Raw Sensory Input in Embodied AI Systems

Manasy Jayasurya
article en

Abstract

Language produced by a system that has never acted in the world describes that world from the outside. Embodied systems that perceive through their own sensors and change what they perceive by acting offer a different setting for language generation, one in which the meaning of a term can be tied to sensory evidence and to the actions the term licenses. This article examines what such grounding requires and how far current systems supply it. We distinguish three requirements, namely perceptual anchoring of reference, action conditioned meaning and situated deixis, and show that pipelines which caption an image and pass the caption to a planner satisfy only the first. End-to-end architectures that map sensory observations and instructions directly to actions and to language remove the intermediate text bottleneck, and we compare eight such systems by input, output and training signal. We then examine the data available, which is smaller than the corpora used for text pre-training by five orders of magnitude, and the evaluation problem, where reference based caption metrics reward fluency and cannot detect a description of an object that is not present. A taxonomy of grounding failures is proposed together with the diagnostic each one requires, and seven open problems are stated.

Eduschool Journal of Artificial Intelligence Research (EJAIR)Vol. 1(3)
Multimodal Machine Learning Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon