A multimodal transformer model for fish vitality prediction using time-series water temperature data and underwater videos

This paper deals with fish vitality prediction using the proposed deep-learning pipeline framework based on water temperature data and synchronized underwater video segments. Fish behaviors indicate welfare and health but deploying an aquaculture Internet of Things (AIoT) system to observe them is challenging. Given a time window t, we design a multimode sensing system to capture the water temperature sequence of the target environment, the underwater RGB fish video, and a sonar video simultaneously. A deep learning-based optical flow model captures the optical flow maps of individual frames of the RGB video, where each map represents a snapshot of the swimming behaviors of the target fish school. With each optical flow map, the swimming speed of the fish school is computed to represent the video as a sequence of swimming speeds, which is then cascaded with the sequence of water temperature. With the resulting sequence as the input, a transformer-like neural network is trained to synthesize a feature sequence for annotating the fish vitality level of the next time window using a classifier. The predicted fish vitality is finally used to trigger a warning message to send to the user. Our system, based on the sonar video, uses a 3D point cloud reconstruction method to estimate the distribution of swimming positions of fish in the target environment, which is augmented to the warning message for completing the fish vitality report. Experimental results show that timely warnings when sea temperatures are low can reduce fish death losses by 50%.