Transferring Dense Pose to Proximal Animal Classes

Abstract

We consider the problem of dense pose labelling in animal classes. We show that, for proximal to humans classes such as chimpanzees (left), we can obtain excellent performance by learning an integrated recognition architecture from existing data sources, including DensePose for humans as well as detection and segmentation information from other COCO classes (right). The key is to establish a common reference (middle), which we obtain via alignment of the reference models of the animals. This enables training a model for the target class without having to label a single example image for it.

A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives

“…Approaches based on graph neural networks (Scarselli et al, 2009) can encode priors about the observed structure and model correlations between individual keypoints and across time (Cai et al, 2019). For some applications (like modeling soft tissue or Primer volume), full surface reconstructions are needed, and this area has seen tremendous progress in recent years (G€ uler et al, 2018;Sanakoyeu et al, 2020;Zuffi et al, 2019). Such advances can be closely watched and incorporated in neuroscience, but we also believe our field (neuroscience) is ready to innovate in this domain too.…”

“…This empirically highlights why pose estimation is a great summary of such video data. Which keypoints should be extracted, of course, dramatically depends on the model organism and the goal of the study (e.g., many are required for dense, 3D models) (G€ uler et al, 2018;Sanakoyeu et al, 2020;Zuffi et al, 2016), whereas a single point can suffice for analyzing some behaviors . One of the great advantages of deep learning-based methods is that they are very flexible, and the user can define what should be tracked.…”

A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives

“…Innovations in the field of object recognition and detection affect all aforementioned parts of the algorithm, as we discussed already in the context of using pre-trained represen- (167). For some applications (like modeling soft tissue or volume) full surface reconstructions are needed and this area has seen tremendous progress in recent years (12,14,168). Such advances can be closely watched and incorporated in neuroscience, but we also believe our field (neuroscience) is ready to innovate in this domain too.…”

“…This also has the opportunity of shifting the focus in computer vision research: Instead of "only" doing human pose estimation, researchers probably will start evaluating on datasets directly relevant to neuroscience community. Indeed there has been a recent interest in animal-related work at top machine learning conferences (14,169), and providing proper benchmarks for such approaches would be ideal.…”

“…This empirically highlights why pose estimation is a great summary of such video data. Which keypoints should be extracted, of course, dramatically depends on the model organism and the goal of the study; e.g., many are required for dense, 3D models (12)(13)(14), while a single point can suffice for analyzing some behaviors (10). One of the great advantages of deep learning based methods is that they are very flexible, and the user can define what should be tracked.…”