Shared Encoding and Representation of Tool-Mediated and Direct Touch Via Superposition Mechanisms

This study proposes a multimodal predictive neural network model that simulates tool-mediated object perception, addressing the neurologically observed phenomenon where subjective experiences during tool use are similar to direct tactile perception. To explain this phenomenon, we apply a superposition mechanism that processes experiences from both tool use and non-tool use scenarios indistinguishably. Our model processes visual, tactile, and proprioceptive inputs from a simulated robotic arm using an encoder-decoder architecture with a shared Long Short-Term Memory (LSTM) module. Trained on long-term sensorimotor data collected from robot movements during tool use and non-tool use, the network demonstrates the ability to predict visual and tactile sensations even when certain sensory inputs are masked. Notably, our results reveal that tactile stimuli in similar situations are processed uniformly, regardless of tool use. When the model's internal state during tool use is employed to simulate non-tool experiences, it predicts as if the tool were an extension of the arm itself. This suggests that the model has incorporated the tool as an integral part of its body representation, mirroring the phenomenon of tool embodiment observed in humans.