Multimodal behavior recognition for dairy cow digital twin construction under incomplete modalities: A modality mapping completion network approach

The recognition of dairy cow behavior is essential for enhancing health management, reproductive efficiency, production performance, and animal welfare. This paper addresses the challenge of modality loss in multimodal dairy cow behavior recognition algorithms, which can be caused by sensor or video signal disturbances arising from interference, harsh environmental conditions, extreme weather, network fluctuations, and other complexities inherent in farm environments. This study introduces a modality mapping completion network that maps incomplete sensor and video data to improve multimodal dairy cow behavior recognition under conditions of modality loss. By mapping incomplete sensor or video data, the method applies a multimodal behavior recognition algorithm to identify five specific behaviors: drinking, feeding, lying, standing, and walking. The results indicate that, under various comprehensive missing coefficients (λ), the method achieves an average accuracy of 97.87 % ± 0.15 %, an average precision of 95.19 % ± 0.4 %, and an average F1 score of 94.685 % ± 0.375 %, with an overall accuracy of 94.67 % ± 0.37 %. This approach enhances the robustness and applicability of cow behavior recognition based on multimodal data in situations of modality loss, resolving practical issues in the development of digital twins for cow behavior and providing comprehensive support for the intelligent and precise management of farms. • Modality Mapping Completion Network addresses multimodal data loss challenges. • Achieves 97.87 % accuracy and high robustness under varying data loss conditions. • Uses Transformers for effective cross-modal fusion of image and sensor data. • Adapts to real-world challenges like interference, occlusion, and harsh environments.