Task Programming: Learning Data Efficient Behavior Representations

Sun, Jennifer J.; Kennedy, Ann; Zhan, Eric; Anderson, David J.; Yue, Yisong; Perona, Pietro

doi:10.1109/cvpr46437.2021.00290

Cited by 27 publications

(29 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Second, Selfee is a fully unsupervised method developed to annotate animal behaviors. Although some other techniques also adopt semi-supervised or unsupervised learning, they usually require manually labeled pre-defined key points of the images ( Huang et al, 2021 ; Luxem et al, 2020 ); some methods also require expert-defined programs for better performance ( Sun et al, 2021 ). Key point selection and program incorporation require a significant amount of prior knowledge and are subject to human bias.…”

Section: Resultsmentioning

confidence: 99%

“…Another direction will be explainable behavior forecasting for a deeper understanding of animal behaviors. For a long time, behavior forecasting has been a field with extensive investigations in which RNNs, LSTMs, or transformers are usually applied ( Aksan et al, 2020 ; Fragkiadaki, 2015 ; Sun et al, 2021 ). However, most of these works use coordinates of key points as inputs.…”

Section: Discussionmentioning

confidence: 99%

“…For training, 120 frames of a mating behavior video were labeled manually, and 85% were used as the training set. Marked body parts included nose, ears, body center, hips, and bottom, following previous publications ( Segalin et al, 2020 ; Sun et al, 2021 ). The model (ResNet-50 as the backbone) was trained for 100,000 iterations, with a batch size of 16.…”

Section: Methodsmentioning

confidence: 99%

“…Those features are usually sufficient to represent animal behaviors. However, it has been demonstrated that the key points generated by pose estimation are less effective for direct behavior classification or two-dimensional visualization ( Luxem et al, 2020 ; Sun et al, 2021 ). Sophisticated post-processing like recurrent neural networks (RNNs) ( Luxem et al, 2020 ), non-locomotor movement decomposition ( Huang et al, 2021 ), or feature engineering ( Sun et al, 2021 ) can be applied to transform the key points into higher-level discriminative features.…”

Section: Introductionmentioning

confidence: 99%

“…Additionally, the neglection of body parts could cause problems. For example, the position of the proboscis of a fly is commonly neglected in behavior studies using pose estimation software ( Calhoun et al, 2019 ; Sun et al, 2021 ). Still, it is crucial for feeding ( Zhou et al, 2019 ), licking behavior during courtship ( Mezzera et al, 2020 ), and hardness detection for a substrate ( Zhang et al, 2020 ).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Selfee, self-supervised features extraction of animal behaviors

Jia

Guo

et al. 2022

eLife

View full text Add to dashboard Cite

Fast and accurately characterizing animal behaviors is crucial for neuroscience research. Deep learning models are efficiently used in laboratories for behavior analysis. However, it has not been achieved to use an end-to-end unsupervised neural network to extract comprehensive and discriminative features directly from social behavior video frames for annotation and analysis purposes. Here, we report a self-supervised feature extraction (Selfee) convolutional neural network with multiple downstream applications to process video frames of animal behavior in an end-to-end way. Visualization and classification of the extracted features (Meta-representations) validate that Selfee processes animal behaviors in a way similar to human perception. We demonstrate that Meta-representations can be efficiently used to detect anomalous behaviors that are indiscernible to human observation and hint in-depth analysis. Furthermore, time-series analyses of Meta-representations reveal the temporal dynamics of animal behaviors. In conclusion, we present a self-supervised learning approach to extract comprehensive and discriminative features directly from raw video recordings of animal behaviors and demonstrate its potential usage for various downstream applications.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Selfee, self-supervised features extraction of animal behaviors

Jia

Guo

et al. 2022

eLife

View full text Add to dashboard Cite

show abstract

Masked Autoencoder Pretraining for Event Classification in Elite Soccer

Rudolph,

Brefeld

2024

Communications in Computer and Information Science

View full text Add to dashboard Cite

Elucidating the Hierarchical Nature of Behavior with Masked Autoencoders

Stoffl,

Bonnetto,

d’Ascoli

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Natural behavior is hierarchical. Yet, there is a paucity of benchmarks addressing this aspect. Recognizing the scarcity of large-scale hierarchical behavioral benchmarks, we create a novel synthetic basketball playing benchmark (Shot7M2). Beyond synthetic data, we extend BABEL into a hierarchical action segmentation benchmark (hBABEL). Then, we develop a masked autoencoder framework (hBehaveMAE) to elucidate the hierarchical nature of motion capture data in an unsupervised fashion. We find that hBehaveMAE learns interpretable latents on Shot7M2 and hBABEL, where lower encoder levels show a superior ability to represent fine-grained movements, while higher encoder levels capture complex actions and activities. Additionally, we evaluate hBehaveMAE on MABe22, a representation learning benchmark with short and long-term behavioral states. hBehaveMAE achieves state-of-the-art performance without domain-specific feature extraction. Together, these components synergistically contribute towards unveiling the hierarchical organization of natural behavior. Models and benchmarks are available at https://github.com/amathislab/BehaveMAE.

show abstract

Task Programming: Learning Data Efficient Behavior Representations

Cited by 27 publications

References 33 publications

Selfee, self-supervised features extraction of animal behaviors

Selfee, self-supervised features extraction of animal behaviors

Masked Autoencoder Pretraining for Event Classification in Elite Soccer

Elucidating the Hierarchical Nature of Behavior with Masked Autoencoders

Contact Info

Product

Resources

About