A Joint Learning of Force Feedback of Robotic Manipulation and Textual Cues for Granular Materials Classification

Granular materials (GMs) are formed by a collection of particles. Even if their visual representation is straightforward, it can be seriously affected in the visually constrained environment. Based on frequency features observed in force signals, this paper proposes a non-visual classifier, <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GmClass</b>, using the force feedback in the robot-granules interaction. Specifically, we transform the force sequences into the frequency domain and integrate them with high-dimensional textual information into a two-branch architecture for multimodal supervised contrastive learning (MSCL). This method achieves an 84.10% classification accuracy, surpassing traditional supervised learning by 10% and outperforming supervised contrastive learning by more than 40%, demonstrating the positive impact of adding text modality on classification, and when applied to a larger dataset, it attains an even higher 85.28% accuracy, further validating its effectiveness. Also, we demonstrate the performance of our approach in handling unseen particles and the generalization capability for varying data collection parameters. Videos, datasets, and codes are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://sites.google.com/view/gmwork2/ftlearning</uri>.