Multimodal Fusion of Speech and Action Recognition for IoT Device

Developments in deep learning techniques have opened up novel possibilities in the multimodal data fusion field. However, there is a significant gap in the capability of deep learning algorithms to handle multimodal data in real time across various modalities. This work aims to develop a model that integrates deep learning, computer vision, and speech recognition methods to accurately interpret human commands conveyed through gestures and voice in real time. The gesture identification problem was modelled using the Long-Short Term Memory (LSTM) networks. This model was trained on key points that are extracted from landmarks recognized using the MediaPipe architecture. Similarly, Speech recognition utilises the Short-term Fourier transform to generate a spectrogram, which is then inputted into Convolutional Neural Networks (CNNs) to parallelly analyze and recognize spoken commands from audio recordings. The models are combined utilizing a late fusion strategy to make a united decision. The proposed methodology achieved above 95 % accuracy for both speech and gesture recognition modal.