Multimodal system for recording individual-level behaviors in songbird groups

The implicit goal of longitudinal observations of animal groups is to identify individuals and to reliably detect their behaviors, including their vocalizations. Yet, to segment fast behaviors and to extract individual vocalizations from sound mixtures remain challenging problems. Promising approaches are systems that record behaviors with multiple cameras, microphones, and animal-borne wireless sensors. Good systems extract from such multimodal signals perfectly synchronized data streams. In this vein, we designed a modular system (BirdPark) for simultaneously recording small animals wearing custom low-power frequency-modulated radio transmitters. Our custom software-defined radio receiver makes use of a multi-antenna demodulation technique that eliminates data losses due to radio signal fading and that increases the signal-to-noise ratio of the received radio signals by 6.5 dB compared to demodulation from the best single-antenna. Digital acquisition of all data streams relies on a single clock, which offers accurate cross-modal redundancies that can be used to dissect rapid behaviors on time scales well below the video frame period. We demonstrate this feat by reconstructing the wing-stroke phases of free-flying songbirds and by separating the vocalizations among up to eight vocally interacting birds. In conclusion, our work paves the way for automatically dissecting complex social behaviors.