Affordance-driven symbol network construction via large language models

Abstract To enable robots to coexist with humans in dynamic real-world environments, they must possess both autonomy and versatility. Such autonomy requires the ability to interpret situations and select appropriate actions, which goes beyond simple image recognition. Humans rely on common sense and “affordances”—perceived action possibilities in a given context—to determine behavior. For example, the presence of an apple on a plate may lead to different reactions depending on its placement. While recent advancements in object detection models like YOLO have enabled AI to recognize and relate objects, these systems lack affordance information and thus struggle with situational understanding. Affordances are a form of implicit knowledge closely tied to human common sense, making them difficult for conventional AI to process. However, large language models (LLMs) such as ChatGPT, which have been trained on vast human-generated texts, may possess a form of embedded common sense. Building on this, we propose a method for automatically acquiring affordances from symbols using LLM outputs. Our method involves three steps: generating descriptive text via LLMs, analyzing it through morphological and dependency parsing to reconstruct a symbol network, and then calculating affordances based on the network’s structure. An initial experiment using the object “apple” demonstrated the ability to extract scene-specific affordances with high explainability. Ultimately, this approach enables affordance re-cognition in a manner similar to human reasoning.