DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning

Abstract

Quantitative behavioral measurements are important for answering questions across scientific disciplines—from neuroscience to ecology. State-of-the-art deep-learning methods offer major advances in data quality and detail by allowing researchers to automatically estimate locations of an animal’s body parts directly from images or videos. However, currently available animal pose estimation methods have limitations in speed and robustness. Here, we introduce a new easy-to-use software toolkit, DeepPoseKit, that addresses these problems using an efficient multi-scale deep-learning model, called Stacked DenseNet, and a fast GPU-based peak-detection algorithm for estimating keypoint locations with subpixel precision. These advances improve processing speed >2x with no loss in accuracy compared to currently available methods. We demonstrate the versatility of our methods with multiple challenging animal pose estimation tasks in laboratory and field settings—including groups of interacting individuals. Our work reduces barriers to using advanced tools for measuring behavior and has broad applicability across the behavioral sciences.

Deep learning tools for the measurement of animal behavior in neuroscience

“…This is an attractive solution for real-time applications, but it is not quite as accurate. For various datasets, DeepPoseKit reports it is about three times as accurate as LEAP, yet similar to DeepLabCut (60). They also report about twice faster video processing compared to DeepLabCut and LEAP for batch-processing (on small frame sizes).…”

“…So far, the toolboxes have been tested on data from the same distribution (i.e. by splitting frames from videos into test and training data), which is important for assessing the performance (48,55,60), but did not directly tested out-of-domain robustness.…”

OpenMonkeyStudio: Automated Markerless Pose Estimation in Freely Moving Macaques

“…For instance, DeepLabCut leverages a pre-trained deep learning model (based on ImageNet) to accurately localize body landmarks. These methods work for various organisms like flies, worms, and mice by learning from a larger number of images collected from a single view (17)(18)(19)(20)(21)(22). However, macaques present several problems that make current best markerless motion capture unworkable.…”

DeepLabStream: Closing the loop using deep learning-based markerless, real-time posture detection

“…Networks trained on either version can be fully integrated into DLStream and used as needed. Additionally, DLStream is theoretically able to support positional information from other pose estimation networks 2,40 . Recent work from Graving et al 40 suggests that using a different network architecture similar result with faster processing speed can be achieved in offline analysis.…”

“…Additionally, DLStream is theoretically able to support positional information from other pose estimation networks 2,40 . Recent work from Graving et al 40 suggests that using a different network architecture similar result with faster processing speed can be achieved in offline analysis. Whether this can be directly translated into closed-loop performance needs to be tested, especially considering the quality of tracking.…”