Ignoring what we don't know in accelerometer-based behaviour classification: the open-set recognition problem

Supervised machine learning is commonly used to classify fine-scale behaviours from animal-borne accelerometers, assigning new data to predefined behaviour categories seen during training. These models cannot recognise novel behaviours as 'unknown', however, and, when exposed to new behaviours, will continue to overpredict the known classes. This issue - known as open-set recognition - is an inevitable, but underexplored, limitation in accelerometer-based behaviour classification. Here, we describe the problem and assess four solutions: (1) a multiclass model with an 'other' category, (2) threshold-based models, (3) one-class models and (4) binary one-versus-all models. We show that traditional multiclass models produce high false-positive rates when exposed to behaviours not present during training. We instead suggest the implementation of binary one-versus-all models as a more conservative method, particularly in cases where a single or limited set of behaviours are of interest. Awareness of this challenge will enhance recognition of often unreported uncertainty in real-world applications.