3D-MuPPET: 3D Multi-Pigeon Pose Estimation and Tracking

Abstract

Markerless methods for animal posture tracking have been rapidly developing recently, but frameworks and benchmarks for tracking large animal groups in 3D are still lacking. To overcome this gap in the literature, we present 3D-MuPPET, a framework to estimate and track 3D poses of up to 10 pigeons at interactive speed using multiple camera views. We train a pose estimator to infer 2D keypoints and bounding boxes of multiple pigeons, then triangulate the keypoints to 3D. For identity matching of individuals in all views, we first dynamically match 2D detections to global identities in the first frame, then use a 2D tracker to maintain IDs across views in subsequent frames. We achieve comparable accuracy to a state of the art 3D pose estimator in terms of median error and Percentage of Correct Keypoints. Additionally, we benchmark the inference speed of 3D-MuPPET, with up to 9.45 fps in 2D and 1.89 fps in 3D, and perform quantitative tracking evaluation, which yields encouraging results. Finally, we showcase two novel applications for 3D-MuPPET. First, we train a model with data of single pigeons and achieve comparable results in 2D and 3D posture estimation for up to 5 pigeons. Second, we show that 3D-MuPPET also works in outdoors without additional annotations from natural environments. Both use cases simplify the domain shift to new species and environments, largely reducing annotation effort needed for 3D posture tracking. To the best of our knowledge we are the first to present a framework for 2D/3D animal posture and trajectory tracking that works in both indoor and outdoor environments for up to 10 individuals. We hope that the framework can open up new opportunities in studying animal collective behaviour and encourages further developments in 3D multi-animal posture tracking.

Peering into the world of wild passerines with 3D-SOCS: synchronized video capture and posture estimation

“…Our target species was great tits, although a smaller number of tagged blue tits were also present, so we extended the model to detect their keypoints. We adopted a top-down approach similar to [37] by: 1) localizing individual birds in each frame using YOLOv8 [67], 2) estimating 14 unique 2D keypoints (Figure 1C) with DeepLabCut [40,68], and 3) triangulating 2D postures into 3D (Figure 1D). For each unique individual recorded in the study (N = 57 great tits, N = 20 blue tits), we collated all 3D information and calculated the median head and body skeleton, then ensured the detections fit the median head skeletons by computing the estimated rotation translation of the skeleton in 3D space.…”

“…Computer vision enables the automatic quantification of fine-scaled behaviors from images, not only increasing the volume of behavioral data but also enhancing the objectivity of behavior coding. Although initial applications were primarily in controlled laboratory settings [34][35][36][37][38][39][40][41][42][43][44][45][46], computer vision is also being applied in the field. It has been used to monitor animals from drones [47][48][49], quantify courtship displays [50], analyze parental care behaviors [51], study underwater interactions [52], identify individuals [53,54], and more.…”

“…With the recent development of frameworks like DeepLabCut, DeepPoseKit and SLEAP, multi-animal posture estimation in 2D and subsequent automated behavioural analysis have become possible [40,46,48,56]. In 3D, frameworks like Anipose, DANNCE and 3D-MuPPET [37,41,45] have also been developed to estimate postures in 3D using multiple camera views (but see [44] for single view 3D estimation). However, these frameworks are often limited to tracking a single animal, and only a few recent studies have demonstrated the ability to estimate the 3D postures of multiple individuals simultaneously (see [37,57,58]).…”

“…In 3D, frameworks like Anipose, DANNCE and 3D-MuPPET [37,41,45] have also been developed to estimate postures in 3D using multiple camera views (but see [44] for single view 3D estimation). However, these frameworks are often limited to tracking a single animal, and only a few recent studies have demonstrated the ability to estimate the 3D postures of multiple individuals simultaneously (see [37,57,58]). Furthermore, to date, applications in the wild have been limited, with notable exceptions including posture estimation of a single cheetah [59] and of three pigeons in semicaptive outdoor systems [37].…”

“…However, these frameworks are often limited to tracking a single animal, and only a few recent studies have demonstrated the ability to estimate the 3D postures of multiple individuals simultaneously (see [37,57,58]). Furthermore, to date, applications in the wild have been limited, with notable exceptions including posture estimation of a single cheetah [59] and of three pigeons in semicaptive outdoor systems [37]. The lack of adoption of 3D animal posture tracking can be attributed to a few main challenges.…”