Recording session leakage inflates accuracy in marine mammal call classification

Splitting clips at random overstated the accuracy of a marine mammal call classifier by up to 23 percentage points, because clips from the same recording tape landed in both training and test data. I classified 4,761 recordings of nine Arctic and subarctic marine mammal species from the Watkins Marine Mammal Sound Database, recorded on 148 tapes between 1949 and 1997, at two levels: species and taxonomic group (baleen whales, toothed whales, pinnipeds). A random forest on 90 acoustic features, a small transformer trained from scratch on log mel spectrograms and linear classifiers on embeddings from the pretrained BirdNET and Perch 2.0 models were evaluated with 5 fold cross validation grouped by tape, and the random forest and both embedding classifiers were also scored with random splits. With random splits the random forest reached 97% accuracy for both tasks. With whole tapes held out, species accuracy fell to 74% (balanced accuracy 49%, against 89% with random splits), while group accuracy fell to 91% (balanced accuracy 71%). The loss was concentrated in species recorded on few tapes: F1 for narwhal fell from 0.91 to 0.02 and for ringed seal from 0.73 to 0.00, with most of their clips predicted as killer whale. The transformer matched the random forest on balanced accuracy for groups (71%) but trailed it for species (42%). Embeddings from the pretrained BirdNET and Perch 2.0 models did best on unseen tapes (species balanced accuracy 66% and 69%) but were still inflated by 28 and 26 points under random splits. Unsupervised clustering recovered neither species nor groups. Random splits of clip level bioacoustic data therefore measure partly how well a model recognises recording conditions, not animals, even when the model is pretrained. I recommend grouping cross validation folds by recording session, reporting balanced accuracy and per class scores, and reporting the random split score alongside as a measure of leakage.

Authors

Publication Details

Published
2026-11-13
DOI
https://doi.org/10.32942/x2p68x
Primary Topic
Marine animal studies overview
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Recording session leakage inflates accuracy in marine mammal call classification

Erzsebet Frey
3 links
Marine animal studies overview
preprint

Recording session leakage inflates accuracy in marine mammal call classification

Erzsebet Frey
preprint en

Abstract

Splitting clips at random overstated the accuracy of a marine mammal call classifier by up to 23 percentage points, because clips from the same recording tape landed in both training and test data. I classified 4,761 recordings of nine Arctic and subarctic marine mammal species from the Watkins Marine Mammal Sound Database, recorded on 148 tapes between 1949 and 1997, at two levels: species and taxonomic group (baleen whales, toothed whales, pinnipeds). A random forest on 90 acoustic features, a small transformer trained from scratch on log mel spectrograms and linear classifiers on embeddings from the pretrained BirdNET and Perch 2.0 models were evaluated with 5 fold cross validation grouped by tape, and the random forest and both embedding classifiers were also scored with random splits. With random splits the random forest reached 97% accuracy for both tasks. With whole tapes held out, species accuracy fell to 74% (balanced accuracy 49%, against 89% with random splits), while group accuracy fell to 91% (balanced accuracy 71%). The loss was concentrated in species recorded on few tapes: F1 for narwhal fell from 0.91 to 0.02 and for ringed seal from 0.73 to 0.00, with most of their clips predicted as killer whale. The transformer matched the random forest on balanced accuracy for groups (71%) but trailed it for species (42%). Embeddings from the pretrained BirdNET and Perch 2.0 models did best on unseen tapes (species balanced accuracy 66% and 69%) but were still inflated by 28 and 26 points under random splits. Unsupervised clustering recovered neither species nor groups. Random splits of clip level bioacoustic data therefore measure partly how well a model recognises recording conditions, not animals, even when the model is pretrained. I recommend grouping cross validation folds by recording session, reporting balanced accuracy and per class scores, and reporting the random split score alongside as a measure of leakage.

Marine animal studies overview
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon