Bioacoustic raw-audio transformer effectively and efficiently encodes mixed sample rate data

Abstract Deep learning has been widely adopted in the field of bioacoustics, enabling behavioral and ecological research into many sound producing animals through the use of specially designed and trained models. So-called foundational models can generalize to novel species or tasks, but require larger and more diverse training datasets. But the broad range of frequencies that animals produce poses a problem: bioacoustic datasets are recorded at various sample rates to capture different frequency ranges. Combining these data to train generalist models becomes exceedingly costly the wider the frequency range is expanded with high resolution data, becoming untenable for bioacousticians. To approach this problem, we performed experiments using a state-of-the-art transformer model, animal2vec, and a custom dataset consisting of recordings with various sample rates and species whose vocalizations cover a broad frequency range. We designed a positional encoding that provides inductive information about the sample rate, but found that the model is natively able to compensate for the perceived time stretch in the receptive field induced by varying sample rates, with no effect on performance compared to the standard approach of upsampling. This opens up new possibilities of training powerful models on broad bioacoustic datasets and the creation of more generalizable tools for bioacousticians.