Deep learning-based behavioral analysis reaches human accuracy and is capable of outperforming commercial solutions

Abstract

To study brain function, preclinical research heavily relies on animal monitoring and the subsequent analyses of behavior. Commercial platforms have enabled semi high-throughput behavioral analyses by automating animal tracking, yet they poorly recognize ethologically relevant behaviors and lack the flexibility to be employed in variable testing environments. Critical advances based on deep-learning and machine vision over the last couple of years now enable markerless tracking of individual body parts of freely moving rodents with high precision. Here, we compare the performance of commercially available platforms (EthoVision XT14, Noldus; TSE Multi-Conditioning System, TSE Systems) to cross-verified human annotation. We provide a set of videos—carefully annotated by several human raters—of three widely used behavioral tests (open field test, elevated plus maze, forced swim test). Using these data, we then deployed the pose estimation software DeepLabCut to extract skeletal mouse representations. Using simple post-analyses, we were able to track animals based on their skeletal representation in a range of classic behavioral tests at similar or greater accuracy than commercial behavioral tracking systems. We then developed supervised machine learning classifiers that integrate the skeletal representation with the manual annotations. This new combined approach allows us to score ethologically relevant behaviors with similar accuracy to humans, the current gold standard, while outperforming commercial solutions. Finally, we show that the resulting machine learning approach eliminates variation both within and between human annotators. In summary, our approach helps to improve the quality and accuracy of behavioral data, while outperforming commercial systems at a fraction of the cost.

Deep-learning based identification, tracking, pose estimation, and behavior classification of interacting primates and mice in complex environments

“…We first examined the performance of SIPEC:SegNet on top-view video recordings of individual mice, behaving in an open-field test (OFT). 8 mice were freely behaving for 10 minutes in the TSE Multi Conditioning System’s OFT arena, previously described in Sturman et al 13 . We labeled the outlines of mice in a total of 23 frames using the VGG image annotator 20 from videos of randomly selected mice.…”

“…The increased performance with fewer labels comes at the cost of a higher computational demand since we increased the dimensionality of the input data by several orders of magnitude (12 pose estimates vs. 16384 pixels). To test our performance we used the data and labels from Sturman et al 13 of 20 freely behaving mice in an OFT. The behavior of these mice was independently annotated by 3 different researchers on a frame-by-frame basis using the VGG video annotation tool 20 .…”

“…Annotations included the following behaviors: supported rears, unsupported rears, grooming and none (unlabeled default class). While Sturman et al 13 evaluated the performance of their behavioral event detection by averaging across chunks of time, evaluating the frame-by-frame performance is more suitable for testing the actual network performance since it was trained the same way. Doing such frame-by-frame analysis shows that SIPEC:BehaveNet has fewer false positives as well as false negatives with respect to the DLC-based approach of Sturman et al We illustrate a representative example of the performance of both approaches for each of the behaviors with their respective ground truths (Figure 3a).…”

DeepEthogram: a machine learning pipeline for supervised behavior classification from raw pixels

“…In addition, it is possible that specialized pipelines designed specifically for one task, such as mouse social behaviors 7 , could perform better at that specific task. An alternate approach to ours is to use innovative methods for estimating pose, including DeepLabCut 22,23 , LEAP 27 , and others 25 , followed by frame-by-frame classification of behaviors based on pose in a supervised 7,19,21 or unsupervised 17 way. Using pose for classification could make behavior classifiers faster to train and less susceptible to overfitting.…”

“…Pioneering work, including JAABA 18 , SimBA 19 , MARS 7 , and others 7,20,21 , has made important progress toward the goal of supervised classification of behaviors. These methods track specific features of an animal's body and use the time series of these features to classify whether a behavior is present at a given timepoint.…”