Selfee, self-supervised features extraction of animal behaviors

Abstract

Fast and accurately characterizing animal behaviors is crucial for neuroscience research. Deep learning models are efficiently used in laboratories for behavior analysis. However, it has not been achieved to use an end-to-end unsupervised neural network to extract comprehensive and discriminative features directly from social behavior video frames for annotation and analysis purposes. Here, we report a self-supervised feature extraction (Selfee) convolutional neural network with multiple downstream applications to process video frames of animal behavior in an end-to-end way. Visualization and classification of the extracted features (Meta-representations) validate that Selfee processes animal behaviors in a way similar to human perception. We demonstrate that Meta-representations can be efficiently used to detect anomalous behaviors that are indiscernible to human observation and hint in-depth analysis. Furthermore, time-series analyses of Meta-representations reveal the temporal dynamics of animal behaviors. In conclusion, we present a self-supervised learning approach to extract comprehensive and discriminative features directly from raw video recordings of animal behaviors and demonstrate its potential usage for various downstream applications.

ConstrastivePose: A contrastive learning approach for self-supervised feature engineering for pose estimation and behavorial classification of interacting animals

“…The model can then be fine-tuned with small amounts of training data to be optimized for downstream tasks. Recently, contrastive learning has been applied for feature extraction from animal videos by Jia et al[25], by performing contrastive learning on the frame images directly in similar fashion to existing work in computer vision.…”

“…The model can then be fine-tuned with small amounts of training data to be optimized for downstream tasks. Recently, contrastive learning has been applied for feature extraction from animal videos by Jia et al [25], by performing contrastive learning on the frame images directly in similar fashion to existing work in computer vision. Contrastive learning's goal is to learn representations of data such that similar datapoints are close to each other, while dissimilar ones are far apart, without the need for labels [24].…”

CNN-Based Action Recognition and Pose Estimation for Classifying Animal Behavior from Videos: A Survey

“…This section provides a high-level overview of animal behavior classification frameworks for small laboratory animals. We outline a general taxonomy that organizes methods as supervised or unsupervised at a coarse level, and with varying degrees of supervision Taxonomy for Animal Behavior Classification Supervised Classification Hand-crafted Features, Behavior Labels [137], [33], [72], [20], [47], [79], [60], [101], [35], [45] Behavior Labels [91], [160], [63] Hand-crafted Features, Pose and Behavior Labels, PE [139], [145], [146], [4], [94], [121] Pose and Behavior Labels, PE [179] Optical Flow, Hand-crafted Features, Behavior Labels [161] Residual Learning, Optical Flow, Behavior Labels [14] Residual Learning, Pose and Behavior Labels, PE [178] Residual Learning, Optical Flow, Behavior Labels [105] Unsupervised Classification Hand-crafted Features, Pose Labels, PE [61] Fully Unsupervised [144], [11], [172], [8], [16], [73] Fig. 8.…”

Using deep learning to study emotional behavior in rodent models

“…This approach has the benefit of not requiring the experimenter to specify rules for deriving output labels for training, at the cost of potentially making the process more sensitive to noise and occlusions (Hausmann et al, 2021 ; Luxem et al, 2022 ). Selfee (Jia et al, 2022 ) and BehaveNet (Batty et al, 2019 ), unlike VAME and DBM, operate directly on snippets of video rather than pose data from DLC/SLEAP, employing autoencoder-style training directly on video data for nonlinear dimensionality reduction. Selfee operates on short 3-frame snippets of raw video, has been used in mice to identify behaviors such as social nose contact and allogrooming in open field tests, and places its main emphasis on using the extracted feature space for a variety of downstream analyses (Jia et al, 2022 ).…”

“…Selfee (Jia et al, 2022 ) and BehaveNet (Batty et al, 2019 ), unlike VAME and DBM, operate directly on snippets of video rather than pose data from DLC/SLEAP, employing autoencoder-style training directly on video data for nonlinear dimensionality reduction. Selfee operates on short 3-frame snippets of raw video, has been used in mice to identify behaviors such as social nose contact and allogrooming in open field tests, and places its main emphasis on using the extracted feature space for a variety of downstream analyses (Jia et al, 2022 ). BehaveNet, unlike the other self-supervised methods, performs feature extraction on individual frames rather than sequences of frames, and does not consider the temporal structure of the data until the discretization step, which uses an autoregressive hidden Markov model (Batty et al, 2019 ).…”