AI-Aided Non-Visual Object Sensing and Replicating for Digital Transformations

This paper presents a vibration-acoustic-based material classification system for material-aware digital twin construction using off-the-shelf smart mobile devices. While existing digital twin research predominantly employs visual sensing (cameras, LiDAR) for geometric representation, these approaches cannot determine material composition when properties are not visually discernible. We propose a novel framework leveraging mobile device’s actuators to generate controlled vibrations and capturing acoustic signatures via the built-in microphone. Acoustic data undergoes feature extraction through time-domain vectorization and Mel-spectrogram representation, enabling classification via SVM, MLP, and CNN architectures. Our CNN model achieves 91% accuracy in controlled environments and 75.4% in real-world scenarios using 1.0-second audio segments, with inference latency below 25 milliseconds. These results validate acoustic sensing effectiveness for material classification, enabling material-aware digital twin construction through on-device processing without additional hardware requirements.