Creatures Great and SMAL: Recovering the Shape and Motion of Animals from Video

Abstract

We present a system to recover the 3D shape and motion of a wide variety of quadrupeds from video. The system comprises a machine learning front-end which predicts candidate 2D joint positions, a discrete optimization which finds kinematically plausible joint correspondences, and an energy minimization stage which fits a detailed 3D model to the image. In order to overcome the limited availability of motion capture training data from animals, and the difficulty of generating realistic synthetic training images, the system is designed to work on silhouette data. The joint candidate predictor is trained on synthetically generated silhouette images, and at test time, deep learning methods or standard video segmentation tools are used to extract silhouettes from real data. The system is tested on animal videos from several species, and shows accurate reconstructions of 3D shape and pose.

Multi-animal pose estimation and tracking with DeepLabCut

“…Optionally (and only when analyzing videos), affinity costs between body parts can be weighted so as to prioritize strong connections that were preferentially selected in the past frames. To this end, and inspired by (40), we compute a temporal coherence cost as follows: where γ controls the influence of distant frames (and is set to 0.01 by default); c and c n are the current connection and its closest neighbor in the relevant past frame; and Δ t is the temporal gap separating these frames.…”

“…Optionally (and only when analyzing videos), affinity costs between body parts can be weighted so as to prioritize strong connections that were preferentially selected in the past frames. To this end, and inspired by (40), we compute a temporal coherence cost as follows:…”

Deep learning tools for the measurement of animal behavior in neuroscience

“…Typically, large datasets are collected to enable the creation of robust algorithms for inference on diverse humans (or for animals, scanning toy models has been fruitful (44)). Recently, outstanding improvements have been made to capture shapes of animals from images (22,45,46). However, there are no animal-specific toolboxes geared towards neuroscience applications, although we believe that this will change in the near future, as for many applications having the soft-tissue measured will be highly important, i.e.…”

MacaquePose: A Novel “In the Wild” Macaque Monkey Pose Dataset for Markerless Motion Capture

“…The performance evaluation of the network trained with the present dataset revealed that there is still room for improvement regarding the misattribution of the limb keypoints (Figure 2h, Table 2), although the RMSE indicates the humanlevel performance (Figure 3). The DeepLabCut algorithm (Mathis et al, 2018) used in the present evaluation does not explicitly utilize the prior knowledge about the animal's body, whereas the other algorithms were suggested to use the connection between keypoints (Insafutdinov et al, 2016;Cao et al, 2017) or 3D shape of the subject (Biggs et al, 2018;Zuffi et al, 2019). Such utilization of the prior knowledge may help to improve the estimation.…”