Dual Autoencoder-Based Joint Learning: Enhancing Depth Classification of Hard Inclusions in Soft Tissue for Robotic Palpation

In robot-assisted minimally invasive surgery (RMIS), robotic palpation is vital for enhancing tissue assessment accuracy, especially for tumor depth detection, which is crucial for precise resections and improved treatment outcomes. By complementing and leveraging the strengths of data from multiple tactile sensors, the tumor detection task can realize significant improvements in accuracy and reliability, thereby enhancing overall performance in robotic palpation in RMIS. However, challenges arise due to differences in sensor modalities and the lack of unified data representation. To address this, we fabricate silicone-based phantom tissue to simulate soft tissue and embedded simulated tumors with hard inclusions at depths ranging from 0–11 mm, treating it as a 12-class classification problem. We conduct two robotic palpation experiments: one with the BarrettHand capacitive sensor and the other with the Digit sensor, collecting two tactile datasets. To explore the joint learnability of datasets collected from different tactile sensors, we design a dual autoencoder-based joint learning framework that integrates two recurrent autoencoders to process the two different tactile datasets. By applying a joint loss mechanism to connect their latent spaces, the autoencoders are trained jointly together, with the latent representation used for supervised classification. Extensive experiments show that joint learning enables sharing of features learned from different tactile datasets, thereby enhancing learning efficiency and classification accuracy for tactile datasets with different modalities and improving the performance of both autoencoders compared to independent training.