Three-dimensional unsupervised probabilistic pose reconstruction (3D-UPPER) for freely moving animals

Ebrahimi, Aghileh S.; Orlowska‐Feuer, Patrycja; Huang, Qian; Zippo, Antonio; Martial, Franck P.; Petersen, Rasmus S.; Storchi, Riccardo

doi:10.1038/s41598-022-25087-4

Cited by 7 publications

(3 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some exemplary (relatively) recent applications of (squared) Mahalanobis distance can be found e.g. in Xu et al [301], Sun et al [302], Timmermann et al [303], Wauchope et al [304], Weinberger [305], Wen et al [306], Yang et al [307], Zhang et al [308], Burssens et al [309], Choi et al [310], Choi et al [311], Dahlin et al [312], Ebrahimi et al [313], Jeong et al [314], Kim et al [315], Nowakowski et al [316], Qu et al [317], Rabby et al [318], Sarno et al [319], Tang et al [320], Tsvieli & Weinberger [321], Zhang et al [322], Zhou et al [323]. Some exemplary (relatively) recent applications of generally non-separable (ordinary/classical) Bregman distances appear e.g.…”

Section: A Further Divergences and Friendsmentioning

confidence: 99%

A Precise Bare Simulation Approach to the Minimization of Some Distances. I. Foundations

Broniatowski¹,

Stummer

2023

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

The constrained minimization (respectively maximization) of directed distances and of related generalized entropies is a fundamental task in information theory as well as in the adjacent fields of statistics, machine learning, artificial intelligence, signal processing and pattern recognition. In our previous paper "A precise bare simulation approach to the minimization of some distances. I. Foundations", we obtained such kind of constrained optima by a new dimension-free precise bare (pure) simulation method, provided basically that (i) the underlying directed distance is of f −divergence type, and that (ii) this can be connected to a light-tailed probability distribution in a certain manner. In the present paper, we extend this approach such that constrained optimization problems of a very huge amount of directed distances and generalized entropies -and beyond -can be tackled by a newly developed dimension-free extended bare simulation method, for obtaining both optima as well as optimizers. Almost no assumptions (like convexity) on the set of constraints are needed, within our discrete setup of arbitrary dimension, and our method is precise (i.e., converges in the limit). For instance, we cover constrained optimizations of arbitrary f −divergences, Bregman distances, scaled Bregman distances and weighted ℓr−distances. The potential for wide-spread applicability is indicated, too; in particular, we deliver many recent references for uses of the involved distances/divergences in various different research fields (which may also serve as an interdisciplinary interface).

show abstract

Section: A Further Divergences and Friendsmentioning

confidence: 99%

A Precise Bare Simulation Approach to the Minimization of Some Distances. I. Foundations

Broniatowski¹,

Stummer

2023

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

show abstract

“…Our cameras see three-dimensional bodies from a two-dimensional perspective. It is increasingly common to record behavior using multiple synchronized cameras, train a network to estimate pose independently in each 2D view, and then use standard stereo vision techniques post hoc to fuse those 2D pose predictions into a 3D pose (Mathis et al, 2020;Karashchuk et al, 2021;Zhang et al, 2021;Ebrahimi et al, 2023). This approach has two limitations.…”

Section: Multi-view Consistency Lossmentioning

confidence: 99%

“…20 . 21 We have presented a deep learning system for easy prototyping and scalable training of pose estimation networks using both labeled frames and many unlabeled videos (semi-supervised learning). We have introduced spatiotemporal constraints that capture a set of prior beliefs on postural dynamics.…”

Section: A Cloud-hosted Application For Pose Estimation As a Servicementioning

confidence: 99%

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools

Biderman

Whiteway

Hurwitz

et al. 2023

Preprint

View full text Add to dashboard Cite

Pose estimation algorithms are shedding new light on animal behavior and intelligence. Most existing models are only trained with labeled frames (supervised learning). Although effective in many cases, the fully supervised approach requires extensive image labeling, struggles to generalize to new videos, and produces noisy outputs that hinder downstream analyses. We address each of these limitations with a semi-supervised approach that leverages the spatiotemporal statistics of unlabeled videos in two different ways. First, we introduce unsupervised training objectives that penalize the network whenever its predictions violate smoothness of physical motion, multiple-view geometry, or depart from a low-dimensional subspace of plausible body configurations. Second, we design a new network architecture that predicts pose for a given frame using temporal context from surrounding unlabeled frames. These context frames help resolve brief occlusions or ambiguities between nearby and similar-looking body parts. The resulting pose estimation networks achieve better performance with fewer labels, generalize better to unseen videos, and provide smoother and more reliable pose trajectories for downstream analysis; for example, these improved pose trajectories exhibit stronger correlations with neural activity. We also propose a Bayesian post-processing approach based on deep ensembling and Kalman smoothing that further improves tracking accuracy and robustness. We release a deep learning package that adheres to industry best practices, supporting easy model development and accelerated training and prediction. Our package is accompanied by a cloud application that allows users to annotate data, train networks, and predict new videos at scale, directly from the browser.

show abstract

3D-MuPPET: 3D Multi-Pigeon Pose Estimation and Tracking

Waldmann,

Chan,

Naik

et al. 2024

Int J Comput Vis

View full text Add to dashboard Cite

Markerless methods for animal posture tracking have been rapidly developing recently, but frameworks and benchmarks for tracking large animal groups in 3D are still lacking. To overcome this gap in the literature, we present 3D-MuPPET, a framework to estimate and track 3D poses of up to 10 pigeons at interactive speed using multiple camera views. We train a pose estimator to infer 2D keypoints and bounding boxes of multiple pigeons, then triangulate the keypoints to 3D. For identity matching of individuals in all views, we first dynamically match 2D detections to global identities in the first frame, then use a 2D tracker to maintain IDs across views in subsequent frames. We achieve comparable accuracy to a state of the art 3D pose estimator in terms of median error and Percentage of Correct Keypoints. Additionally, we benchmark the inference speed of 3D-MuPPET, with up to 9.45 fps in 2D and 1.89 fps in 3D, and perform quantitative tracking evaluation, which yields encouraging results. Finally, we showcase two novel applications for 3D-MuPPET. First, we train a model with data of single pigeons and achieve comparable results in 2D and 3D posture estimation for up to 5 pigeons. Second, we show that 3D-MuPPET also works in outdoors without additional annotations from natural environments. Both use cases simplify the domain shift to new species and environments, largely reducing annotation effort needed for 3D posture tracking. To the best of our knowledge we are the first to present a framework for 2D/3D animal posture and trajectory tracking that works in both indoor and outdoor environments for up to 10 individuals. We hope that the framework can open up new opportunities in studying animal collective behaviour and encourages further developments in 3D multi-animal posture tracking.

show abstract

Three-dimensional unsupervised probabilistic pose reconstruction (3D-UPPER) for freely moving animals

Cited by 7 publications

References 37 publications

A Precise Bare Simulation Approach to the Minimization of Some Distances. I. Foundations

A Precise Bare Simulation Approach to the Minimization of Some Distances. I. Foundations

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools

3D-MuPPET: 3D Multi-Pigeon Pose Estimation and Tracking

Contact Info

Product

Resources

About