Impact of cross-validation designs on cattle behavior prediction using machine learning and deep learning models with tri-axial accelerometer data

Assessing cattle behaviors provides insights into animal health, welfare, and productivity to support on-farm management decisions. Wearable accelerometers offer an alternative approach to traditional human evaluation, providing a more objective and efficient method for predicting cattle behavior. Random cross-validation (CV) is commonly used to evaluate behavior prediction by splitting data into training and testing sets, but it can yield inflated results when records from the same animal are included in both sets. Block CV splits data by block effects, offering a more realistic evaluation but remains underexplored for predicting multi-class imbalanced cattle behavior. Additionally, deep learning (DL) models have not been fully explored for behavior prediction compared to machine learning (ML) models. The objectives of this study were to examine the impact of CV designs on multi-class imbalanced cattle behavior prediction and to compare the performance of ML and DL models. Three ML and two DL models were used to predict the four behaviors of six beef cows from a public tri-axial accelerometer dataset, with model performance evaluated using both hold-out and leave-cow-out CV designs representing random and block CV, respectively. In hold-out CV, ML models achieved accuracies of 0.94 to 0.95 and F1 scores of 0.93 to 0.94, while DL models achieved accuracies of 0.9 to 0.92 and F1 scores of 0.89 to 0.91. In the leave-cow-out CV, ML models obtained accuracies of 0.72 to 0.82 and F1 scores of 0.65 to 0.76, whereas DL models obtained accuracies of 0.76 to 0.82 and F1 scores of 0.64 to 0.76. Generally, ML models outperformed DL models in the hold-out CV, but the multi-layer perceptron DL model demonstrated comparable or superior performance in the leave-cow-out CV. All models performed better with hold-out CV than leave-cow-out CV. Our results suggest that CV designs can affect behavior prediction performance. While a random CV produces seemingly good predictions, these results can be artificially inflated by the data partition. A block CV that strategically partitions data could be a more appropriate design.