A Novel Dataset for Keypoint Detection of quadruped Animals from Images

Banik, Prianka; Lin, Li; Dong, Xishuang

doi:10.48550/arxiv.2108.13958

Cited by 4 publications

(11 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous works explore animal pose estimation for a large range of species in laboratory and life sciences (5-10, 17, 18). Several recent works also appeared in computer vision research, such as larger animal datasets (19)(20)(21), testing cross-domain adaptation in quadrupeds (15), out-of-distribution robustness (including cross species) and transfer learning gains with Horse-10 (22). A few papers have started to put together multi-species datasets with the same number of keypoints (with identically named labels) per animal (such as in AnimalPose and AwA (15,19)).…”

Section: Related Workmentioning

confidence: 99%

“…Several recent works also appeared in computer vision research, such as larger animal datasets (19)(20)(21), testing cross-domain adaptation in quadrupeds (15), out-of-distribution robustness (including cross species) and transfer learning gains with Horse-10 (22). A few papers have started to put together multi-species datasets with the same number of keypoints (with identically named labels) per animal (such as in AnimalPose and AwA (15,19)). However, so far the benefit of merging across datasets with different keypoints, and to train a model to predict a superset of keypoints from all available datasets has not been shown.…”

Section: Related Workmentioning

confidence: 99%

“…Additionally, most (of the small set of) cross-species works have focused on non-laboratory quadrupeds (15,19,22,23), while laboratory mice, who are of great importance in neuroscience and behavioral studies, have received much less attention in terms of cross-species or cross-lab animal pose estimation. To our knowledge, we are the first to provide a method to create a cross laboratory top-view mouse model, paving the way for larger efforts.…”

Section: Related Workmentioning

confidence: 99%

“…Therefore, we combine datasets, as shown in Figure 2, that cover these two broad interest areas. Specifically, we use: StanfordDogs (21,49), AnimalPose (15), Horse30 (22), AcinoSet (17), Badja (50), AwA (19), Openfield (51), TriMouse (52), FST-LDB-EPM-CSI-OFT (53), BlackMice (54), and in-house data.…”

Section: Pre-training Datasetsmentioning

confidence: 99%

“…Each dataset has defined different number of keypoints (and naming). For our work, we used most keypoints from each individual dataset, but some are removed for simplicity in downstream tasks (this choice does not interfere with our goals of demonstrating how to merge datasets and assess zero-shot learning performance).We held out rodents, elephants, and giraffes from the AwA dataset (19) to make an OOD test set. We also defined a main keypoint superset per SuperAnimal model in order to map dissimilarly named keypoints and non-overlapping points to the same space.…”

Section: Pre-training Datasetsmentioning

confidence: 99%

See 4 more Smart Citations

SuperAnimal models pretrained for plug-and-play analysis of animal behavior

Ye¹,

Mathis²,

Mathis³

2022

Preprint

View full text Add to dashboard Cite

Animal pose estimation is critical in applications ranging from life science research, agriculture, to veterinary medicine. Compared to human pose estimation, the performance of animal pose estimation is limited by the size of available datasets and the generalization of a model across datasets. Typically different keypoints are labeled regardless of whether the species are the same or not, leaving animal pose datasets to have disjoint or partially overlapping keypoints. As a consequence, a model cannot be used as a plug-and-play solution across datasets. This reality motivates us to develop panoptic animal pose estimation models that are able to predict keypoints defined in all datasets. In this work we propose a simple yet effective way to merge differentially labeled datasets to obtain the largest quadruped and lab mouse pose dataset. Using a gradient masking technique, so called SuperAnimal-models are able to predict keypoints that are distributed across datasets and exhibit strong zero-shot performance. The models can be further improved by (pseudo) labeled fine-tuning. These models outperform ImageNet-initialized models.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Pre-training Datasetsmentioning

confidence: 99%

Section: Pre-training Datasetsmentioning

confidence: 99%

See 3 more Smart Citations

SuperAnimal models pretrained for plug-and-play analysis of animal behavior

Ye¹,

Mathis²,

Mathis³

2022

Preprint

View full text Add to dashboard Cite

show abstract

Demystifying Unsupervised Semantic Correspondence Estimation

Aygün¹,

Aodha²

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

We explore semantic correspondence estimation through the lens of unsupervised learning. We thoroughly evaluate several recently proposed unsupervised methods across multiple challenging datasets using a standardized evaluation protocol where we vary factors such as the backbone architecture, the pre-training strategy, and the pre-training and finetuning datasets. To better understand the failure modes of these methods, and in order to provide a clearer path for improvement, we provide a new diagnostic framework along with a new performance metric that is better suited to the semantic matching task. Finally, we introduce a new unsupervised correspondence approach which utilizes the strength of pre-trained features while encouraging better matches during training. This results in significantly better matching performance compared to current state-of-the-art methods.

show abstract

SuperAnimal pretrained pose estimation models for behavioral analysis

Ye,

Filippova,

Lauer

et al. 2024

Nat Commun

View full text Add to dashboard Cite

Quantification of behavior is critical in diverse applications from neuroscience, veterinary medicine to animal conservation. A common key step for behavioral analysis is first extracting relevant keypoints on animals, known as pose estimation. However, reliable inference of poses currently requires domain knowledge and manual labeling effort to build supervised models. We present SuperAnimal, a method to develop unified foundation models that can be used on over 45 species, without additional manual labels. These models show excellent performance across six pose estimation benchmarks. We demonstrate how to fine-tune the models (if needed) on differently labeled data and provide tooling for unsupervised video adaptation to boost performance and decrease jitter across frames. If fine-tuned, SuperAnimal models are 10–100× more data efficient than prior transfer-learning-based approaches. We illustrate the utility of our models in behavioral classification and kinematic analysis. Collectively, we present a data-efficient solution for animal pose estimation.

show abstract

A Novel Dataset for Keypoint Detection of quadruped Animals from Images

Cited by 4 publications

References 30 publications

SuperAnimal models pretrained for plug-and-play analysis of animal behavior

SuperAnimal models pretrained for plug-and-play analysis of animal behavior

Demystifying Unsupervised Semantic Correspondence Estimation

SuperAnimal pretrained pose estimation models for behavioral analysis

Contact Info

Product

Resources

About