Integrating audio signals and computer vision for automated detection of sow nursing behavior

Nursing Behavior is a complex and important behavior. While providing essential energy and immunoglobulin for the piglets, the health of sows with high lactation efficiency can be altered. Therefore, effective detection and monitoring of this behavior are essential to support piglet growth, to ensure sow health and to identify potential problems promptly. The nursing act requires active participation from the sow and the piglets. While published research highlights the use of computer vision to detect suckling behavior based on piglet location, its accuracy diminishes when piglets are obscured or crowded together. This issue is exacerbated at night due to reduced image clarity. Given these limitations, it is necessary to explore other key characteristics of nursing behavior. This study leverages the distinctive acoustic pattern of sow grunts by integrating audio-based analysis with visual-based automatic validation of sow postures, achieving a precision of 0.93 and a recall of 0.87. Sound signals were used to detect potential nursing time windows. The computer vision model (YOLOv9) was then employed to check the target sow's posture to filter out interference from other sows' grunts. The average processing time for a 1-hour audio file was 53.9 seconds. By addressing the limitations of computer vision, the method proposed in this study simultaneously achieved notable analytical speed. • Automatically detecting nursing behavior using a sensor-fusion approach incorporating sound and visual information. • By using sound to identify the time windows of nursing behavior, the limitations caused by poor lighting and visual occlusion are effectively overcome. • The proposed method allows for more resource-efficient continuous monitoring of nursing behavior.