Challenges of machine learning model validation using correlated behaviour data: Evaluation of cross-validation strategies and accuracy measures

Abstract

Automated monitoring of the movements and behaviour of animals is a valuable research tool. Recently, machine learning tools were applied to many species to classify units of behaviour. For the monitoring of wild species, collecting enough data for training models might be problematic, thus we examine how machine learning models trained on one species can be applied to another closely related species with similar behavioural conformation. We contrast two ways to calculate accuracies, termed here as overall and threshold accuracy, because the field has yet to define solid standards for reporting and measuring classification performances. We measure 21 dogs and 7 wolves, and find that overall accuracies are between 51 and 60% for classifying 8 behaviours (lay, sit, stand, walk, trot, run, eat, drink) when training and testing data are from the same species and between 41 and 51% when training and testing is cross-species. We show that using data from dogs to predict the behaviour of wolves is feasible. We also show that optimising the model for overall accuracy leads to similar overall and threshold accuracies, while optimizing for threshold accuracy leads to threshold accuracies well above 80%, but yielding very low overall accuracies, often below the chance level. Moreover, we show that the most common method for dividing the data between training and testing data (random selection of test data) overestimates the accuracy of models when applied to data of new specimens. Consequently, we argue that for the most common goals of animal behaviour recognition overall accuracy should be the preferred metric. Considering, that often the goal is to collect movement data without other methods of observation, we argue that training data and testing data should be divided by individual and not randomly.

Limitations of using surrogates for behaviour classification of accelerometer data: refining methods using random forest models in Caprids

“…[ 5 , 26 ]). However, the value of using data split per individual datasets has been highlighted when validating the ability of models to predict behaviour of unobserved individuals [ 28 ]. In this study, we built two model sets, the first splitting the data 60/40 randomly, with data from each individual present in both the training and the validation models, and the other approximately split 60/40 at the individual level, with individuals only in either the training or validation sets.…”

“…Importantly, it is particularly problematic to test the value of domestic surrogates for wild animals if those wild animals cannot be observed for verification. For example, applying the common method for splitting data into training and validation data sets overestimates the accuracy of models when tested on new individuals because the models are validated on individuals also used to train the model [ 28 ].…”

Peptides with Dual Antimicrobial–Anticancer Activity: Strategies to Overcome Peptide Limitations and Rational Design of Anticancer Peptides

“…Thus, it is suggested to use cross-validation, which is a technique used to evaluate the results of a statistical analysis. It is used where the main objective is prediction and need to estimate the accuracy of a model ensuring that they are independent of the partitioning between training and test dataset [ 119 , 120 ].…”

“…This process will be repeated n times and, in every iteration, a different test dataset will be selected, while the remaining data will be used, as mentioned, as a training set. Once the iterations are completed, the accuracy and error are calculated for each of the models produced [ 119 , 120 ].…”

Exploratory study of introducing HPC to non-ICT researchers: institutional strategy is possibly needed for widespread adaption

“…In the beginning, this was done in situ during observation, but in modern times the typical routine is to make video recordings of the behaviour which is analysed later to get quantitative results (examples with various taxa are, e.g. experiments with dogs [3], capuchin monkeys [4], cleaner fish [6], and zebra finches [15]). The possibility of recordings opened the possibility for obtaining a wealth of data, but due to lack of tools, analysis is mostly done with human effort.…”