Deep tracks: Using deep learning and procedurally simulated data for automated vertebrate footprints classification

The study of vertebrate footprints provides useful information on animal behavior, locomotion, and ecology. However, automatically classifying these records using photographs is difficult due to the significant morphological variation in footprints and the lack of readily available labeled datasets. To address this issue, this study developed Deep Tracks, a novel Unity application to procedurally create a dataset of simulated footprint images. Two datasets were used to evaluate the influence and impact of the simulated dataset on real footprint classification: (1) a dataset comprising 40,000 simulated footprints, (2) approximately 1,500 real vertebrate footprints from 10 different vertebrate groups. Both simulated and real footprints belong to the following clades: Mammalia (coyotes, foxes, bears, otters, squirrels, raccoons, deer), avian Dinosauria (turkeys) and non-avian Dinosauria (theropods, sauropods). Convolutional Neural Networks (CNNs) were used to classify the different datasets either from the simulated or real footprints. An initial comparison of five different architectures (DenseNet-121, ResNet-18, ResNet-50, EfficientNet-b0, and InceptionNet-v3) was done using the simulated dataset, with EfficientNet-b0 presenting better metrics results. Seven experimental configurations were designed to evaluate different strategies for incorporating the real data into the model development. The first configuration involved training and testing exclusively on real footprints, without any simulated data. The second configuration trained the model on real data, but tested it on simulated footprints. The third configuration used transfer learning to fine-tune a CNN, initially trained on simulated data, for classifying real footprint images. The remaining four configurations incorporated simulated data into the training process alongside a fixed percentage of real data — 20%, 50%, 80%, or 100%. The application of fine-tuning led to an accuracy improvement of over 30% in classifying real footprints, compared to a CNN trained solely on real data. These results highlight the significance of advanced data augmentation techniques in improving both accuracy and reliability in vertebrate footprint classification, particularly in scenarios with limited real data availability. • Developed Deep Tracks, a Unity tool for simulating vertebrate footprint images. • First approach to classify footprints from 10 vertebrate groups, extinct and extant. • Combining simulated and real data improved footprint classification accuracy. • EfficientNet-b0 outperformed other models classifying simulated footprints (97%). • Transfer learning outperformed other models classifying real footprints (74%).