Super machine learning: improving accuracy and reducing variance of behaviour classification from accelerometry

Abstract

Background: Semi-automating the analyses of accelerometry data makes it possible to synthesize large data sets. However, when constructing activity budgets from accelerometry data, there are many methods to extract, analyse and report data and results. For instance, machine learning is a robust approach to classifying data. We used a new method, super learning, that combines base learners (different machine learning methods) in an optimal manner to achieve overall improved accuracy. Other facets of super learning include the number of behavioural categories to predict, the number of epochs (sample window size) used to split data for training and testing and the parameters on which to train the models. Results:The super learner accurately classified behaviour categories with higher accuracy and lower variance than comparative models. For all models tested, using four behaviours, in comparison with six, achieved higher rates of accuracy. The number of epochs chosen also affected the accuracy with smaller epochs (7 and 13) performing better than longer epochs (25 and 75). Conclusions:Correct model selection, training and testing are imperative to creating reliable and valid classification models. To do so means model fitting must use a wide array of selection criteria. We evaluated a number of these including model, number of behaviours to classify and epoch length and then used a parameter grid search to implement the models. We found that all criteria tested contributed to the models' overall accuracies. Fewer behaviour categories and shorter epoch length improved the performance of all models tested. The super learner classified behaviours with higher accuracy and lower variance than other models tested. However, when using this model, users need to consider the additional human and computational time required for implementation. Machine learning is a powerful method for classifying the behaviour of animals from accelerometers. Care and consideration of the modelling parameters evaluated in this study are essential when using this type of statistical analysis.

Combined use of two supervised learning algorithms to model sea turtle behaviours from tri-axial acceleration data

“…Our results, with a Resheff et al (2014) reported 84.02% accuracy for six behaviours in griffon vultures, Graf et al (2015) reported 94.99% accuracy for six behaviours in Eurasian beavers, Castor fiber, and Nathan et al (2012) reported 90.88% accuracy for seven behaviours in griffon vultures. As in previous comparative studies (Nathan et al, 2012;Ellis et al, 2014;Resheff et al, 2014;Ladds et al, 2017), the Random Forest algorithm performed better than the CART algorithm, with specific improvement being manifest in the precision and recall indices of the behaviours: here, the global accuracy was 5.66 points higher with the Random Forest algorithm in the hawksbill and green turtle model, and 7.2 points higher in the loggerhead turtle model. In both cases, the behaviours that showed some difficulty in identification by the CART algorithm were associated with a greater increase of one, or both, of the indices in the Random Forest model.…”

A comparison of techniques for classifying behavior from accelerometers for two species of seabird

“…For both species, including more than two or three predictor variables gave no significant improvement in classification accuracy. Many other studies, particularly those using machine learning methods, include large numbers of predictor variables (Ladds et al, ; Nathan et al, ). We found that limiting the number of variables greatly reduced analysis time, because files are smaller and models are simpler.…”

“…That classification accuracy was consistently high is perhaps not a surprising result. Many studies have found higher accuracy when only a small number of general behaviors is considered (Hammond, Springthorpe, Walsh, & Berg‐Kirkpatrick, ; Ladds et al, ; Shamoun‐Baranes et al, ). Indeed, the behaviors we considered are readily identifiable in an accelerometer trace using the human eye.…”

“…Coarse‐scale behavior identification, like the approaches demonstrated here, could be a first step in a hierarchical process of identifying fine‐scale behaviors (Leos‐Barajas et al, ). Several studies have been successful in distinguishing general behaviors, like the behaviors identified in this paper, but have been less successful in effectively classifying finer scale behaviors associated with prey capture, prey handling and self‐maintenance (Hammond et al, ; Ladds et al, ; Shamoun‐Baranes et al, ). An initial partitioning into general behavior classes may simplify the process of defining detailed behavior profiles, especially where these behaviors occur as a subset within more coarse‐scale behavior.…”

“…Widespread adoption of accelerometers to measure animal behavior is inhibited by limited validation, which has contributed to a lack of consensus on analysis methods. A host of methods have been proposed for classifying animal behavior from accelerometer data (Appendix S1), including movement thresholds (Brown et al, 2013;Moreau, Siebert, Buerkert, & Schlecht, 2009;Shamoun-Baranes et al, 2012), histogram analysis (Collins et al, 2015), k-means (KM) cluster analysis (Angel, Berlincourt, & Arnould, 2016;Sakamoto et al, 2009), k-nearest neighbor analysis (Bidder et al, 2014), classification and regression trees (Shamoun-Baranes et al, 2012), neural networks (NN; Nathan et al, 2012;Resheff, Rotics, Harel, Spiegel, & Nathan, 2014), random forests (Bom, Bouten, Piersma, Oosterbeek, & van Gils, 2014;Nathan et al, 2012;Pagano et al, 2017), hidden Markov models (HMM; Leos-Barajas et al, 2016), expectation maximization (EM; Chimienti et al, 2016), and super machine learning (Ladds et al, 2017). At least three custom software applications are available for classifying animal behavior from trained accelerometer data: AcceleRater (Resheff et al, 2014), G-sphere (Wilson et al, 2016), and Ethographer (Sakamoto et al, 2009).…”