Learning secondary tool affordances from human actions using the iCub robot

Tools and other objects offer agents a range of potential actions, commonly referred to as affordances. Each tool is typically designed with a primary purpose in mind-like a hammer’s function to drive nails. However, tools can also serve purposes beyond their original design. These alternative uses represent secondary affordances, extending the tool’s utility beyond its primary intended function. While prior robotics research on affordance perception and learning has primarily focused on primary affordances, our work addresses the less-explored area of learning secondary tool affordances from human partners. Using the iCub robot equipped with three cameras, we observed humans performing actions on twenty objects using four different tools in ways that deviate from their primary purposes. For example, the iCub observed humans using rulers not for measuring but to push, pull, and move objects. In this setting, we constructed a dataset by taking pictures of objects before and after each action is executed. To model secondary affordance learning, we trained three neural networks (ResNet-18, ResNet-50, and ResNet-101) on three prediction tasks using these raw images as input: (1) identifying which tool was used to move an object, (2) predicting the tool with additional action category information, and (3) jointly predicting both the tool and action performed. Our results demonstrate that deep learning architectures enable the iCub robot to successfully predict secondary tool affordances, thereby paving the road for human-robot collaborative object manipulation involving complex affordances. Code and data from this study are available at https://github.com/BosongDing/second_affordance