Multimodal recognition of visually similar objects using images and impact sounds

Classifying small, similar-looking objects is challenging for unimodal systems like image- or sound-based recognition. We propose Acoustic-Visual Similar Object Recognition (AV-SOR), a multimodal approach combining images and impact sounds from objects falling onto a surface. Our AV-SORNet model integrates a transformer-based audio branch and a vision-based image branch, fusing their embeddings for classification. We evaluate 56 visually similar screwdriver bits, where unimodal models struggled with accuracies of less than 90%. Our multimodal AV-SORNet achieved accuracies of above 98%, highlighting the potential of integrating impact sounds with visual data for precise object identification. This approach is promising for applications like part identification, where fine-grained distinctions are crucial.