Anipose: a toolkit for robust markerless 3D pose estimation

Karashchuk, Pierre; Rupp, Katie L.; Dickinson, Evyn S; Walling-Bell, Sarah; Sanders, Elischa; Azim, Eiman; Brunton, Bingni W.; Tuthill, John C.

doi:10.1101/2020.05.26.117325

Cited by 60 publications

(75 citation statements)

References 98 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Below the touchscreen is a plexiglass panel that allows a free field of view from outside towards the space that contains the head (for face and gaze analysis) and shoulders of the animals. This is useful for eye- and body-tracking systems (Mathis et al, 2018; Karashchuk, 2019; Bala et al, 2020; Sheshadri et al, 2020).…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

A Kiosk Station for the Assessment of Multiple Cognitive Domains and Enrichment of Monkeys

Womelsdorf

Thomas

Neumann

et al. 2021

Preprint

View full text Add to dashboard Cite

Background: Nonhuman primates (NHPs) are self-motivated to perform cognitive tasks on touchscreens in their animal housing setting. To leverage this ability, fully integrated hardware and software solutions are needed, that work within housing and husbandry routines while also spanning cognitive task constructs of the Research Domain Criteria (RDoC). New Method: We describe a Kiosk Station (KS-1) that provides robust hardware and software solutions for running cognitive tasks in cage-housed NHPs. KS-1 consists of a frame for mounting flexibly on housing cages, a touchscreen animal interface with mounts for receptables, reward pumps and cameras, and a compact computer cabinet with an interface for controlling behavior. Behavioral control is achieved with a unity3D program that is virtual-reality capable, allowing semi-naturalistic visual tasks to assess multiple cognitive domains. Results: KS-1 is fully integrated into the regular housing routines of monkeys. A single person can operate multiple KS-1s. Monkeys engage with KS-1 at high motivation and cognitive performance levels at high intra-individual consistency. Comparison with Existing Methods: KS-1 is optimized for flexible mounting onto standard apartment cage systems. KS-1 has a robust animal interface with options for gaze/reach monitoring. It has an integrated user interface for controlling multiple cognitive task using a common naturalistic object space designed to enhance task engagement. All custom KS-1 components are open-sourced. Conclusions: KS-1 is a versatile tool for cognitive profiling and enrichment of cage-housed monkeys. It reliably measures multiple cognitive domains which promises to advance our understanding of animal cognition, inter-individual differences and underlying neurobiology in refined, ethologically meaningful behavioral foraging contexts.

show abstract

Section: Resultsmentioning

confidence: 99%

“…2C ). This allows monitoring the animals task engagement in the kiosk from outside the housing room and the recording of up to five synchronized camera views which will allow 3D reconstruction of gaze and reach patterns (Karashchuk, 2019; Sheshadri et al, 2020).…”

Section: Methodsmentioning

confidence: 99%

A Kiosk Station for the Assessment of Multiple Cognitive Domains and Enrichment of Monkeys

Womelsdorf

Thomas

Neumann

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Extending SLEAP to tracking in 3D from multiple views may be another the direction of future work, though existing 3D pose tracking methods that build off of 2D predictions can already be configured take advantage of SLEAP [20].…”

Section: Discussionmentioning

confidence: 99%

SLEAP: Multi-animal pose tracking

Pereira

Tabris

et al. 2020

Preprint

View full text Add to dashboard Cite

The desire to understand how the brain generates and patterns behavior has driven rapid methodological innovation to quantify and model natural animal behavior. This has led to important advances in deep learning-based markerless pose estimation that have been enabled in part by the success of deep learning for computer vision applications. Here we present SLEAP (Social LEAP Estimates Animal Poses), a framework for multi-animal pose tracking via deep learning. This system is capable of simultaneously tracking any number of animals during social interactions and across a variety of experimental conditions. SLEAP implements several complementary approaches for dealing with the problems inherent in moving from single-to multi-animal pose tracking, including configurable neural network architectures, inference techniques, and tracking algorithms, enabling easy specialization and tuning for particular experimental conditions or performance requirements. We report results on multiple datasets of socially interacting animals (flies, bees, and mice) and describe how dataset-specific properties can be leveraged to determine the best configuration of SLEAP models. Using a high accuracy model (<2.8 px error on 95% of points), we were able to track two animals from full size 1024 × 1024 pixel frames at up to 320 FPS. The SLEAP framework comes with a sophisticated graphical user interface, multi-platform support, Colab-based GPU-free training and inference, and complete tutorials available, in addition to the datasets, at sleap.ai.

show abstract

“…3D pose estimation is typically accomplished by triangulating 2-dimensional (2D) poses acquired using multiple camera views and deep network-based markerless keypoint tracking algorithms [5,6,7,8,9,10,11,12,13]. Notably, triangulation requires that every tracked keypoint be visible from at least two synchronized cameras [14] and that each camera is first calibrated by hand [15,16] or, as in DeepFly3D, by solving a non-convex optimization problem [7]. These expectations are expensive and often difficult to meet, particularly in space-constrained experimental systems that also house sensory stimulation devices [1,2,17].…”

Section: Introductionmentioning

confidence: 99%

“…Notably, triangulation requires that every tracked keypoint, be it a joint or other body feature, be visible from at least two synchronized cameras [14] and that each camera be calibrated. This can be done by hand [15, 16] or, by solving a non-convex optimization problem [7]. These expectations are high and often difficult to meet, particularly in space-constrained experimental systems that also house sensory stimulation devices [1, 2, 17].…”

Section: Introductionmentioning

confidence: 99%

LiftPose3D, a deep learning-based approach for transforming 2D to 3D pose in laboratory animals

Gosztolai¹,

Günel²,

Abrate³

et al. 2020

Preprint

View full text Add to dashboard Cite

Markerless 3D pose estimation has become an indispensable tool for kinematic studies of laboratory animals. Most current methods recover 3D pose by multi-view triangulation of deep network-based 2D pose estimates. However, triangulation requires multiple, synchronised cameras per keypoint and elaborate calibration protocols that hinder its widespread adoption in laboratory studies. Here, we describe LiftPose3D, a deep network-based method that overcomes these barriers by reconstructing 3D poses from a single 2D camera view. We illustrate LiftPose3D’s versatility by applying it to multiple experimental systems using flies, mice, and macaque monkeys and in circumstances where 3D triangulation is impractical or impossible. Thus, LiftPose3D permits high-quality 3D pose estimation in the absence of complex camera arrays, tedious calibration procedures, and despite occluded keypoints in freely behaving animals.

show abstract

Anipose: a toolkit for robust markerless 3D pose estimation

Cited by 60 publications

References 98 publications

A Kiosk Station for the Assessment of Multiple Cognitive Domains and Enrichment of Monkeys

A Kiosk Station for the Assessment of Multiple Cognitive Domains and Enrichment of Monkeys

SLEAP: Multi-animal pose tracking

LiftPose3D, a deep learning-based approach for transforming 2D to 3D pose in laboratory animals

Contact Info

Product

Resources

About