OpenLabCluster: Active Learning Based Clustering and Classification of Animal Behaviors in Videos Based on Automatically Extracted Kinematic Body Keypoints

Li, Jingyuan; Keselman, Moishe Leo; Shlizerman, Eli

doi:10.1101/2022.10.10.511660

Cited by 5 publications

(7 citation statements)

References 70 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We argue that features with behavioral specificity can not be directly extracted in the images using the end-to-end approach, resulting in poor classification performance. Therefore, some studies extracted the pose and tracking data of rats to classify behaviors in an unsupervised way and achieved high accuracies [34, 35]. However, such methods are unable to discriminate specific social behaviors.…”

Section: Resultsmentioning

confidence: 99%

ARBUR, a machine learning-based analysis system for relating behaviors and ultrasonic vocalizations of rats

Chen,

Jia,

Zhou

et al. 2023

Preprint

View full text Add to dashboard Cite

Deciphering how different types of behavior and ultrasonic vocalizations (USVs) of rats interact can yield insights into the neural basis of social interaction. However, the behavior-vocalization interplay of rats remains elusive because of the challenges of relating the two communication media in complex social contexts. Here, we propose a machine learning-based analysis system called ARBUR. ARBUR can cluster without bias both non-step (continuous) and step USVs via three steps, hierarchically detect eight types of behavior of two freely behaving rats with high accuracy, and locate the vocal rat in 3-D space. By simultaneously recording the video and ultrasonic streams of two freely behaving rats, we show that ARBUR can not only automatically reveal the well-understood behavior-associated vocalizations that were carefully concluded by other behavioral researchers, but also hold the promise to indicate novel findings that can be hardly found by manual analysis, especially regarding step USVs and the active/passive rat-associated vocalizations during aggressive/intimate social behaviors. Using ARBUR, we further confirm that rats communicate via distinct USVs when engaging in different types of social behavior. This work could help mechanistically understand the interactive influence between the behavior and USVs of rats.

show abstract

Section: Resultsmentioning

confidence: 99%

ARBUR, a machine learning-based analysis system for relating behaviors and ultrasonic vocalizations of rats

Chen,

Jia,

Zhou

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Therefore, researchers often use a pre-trained deep learning-based model to extract trajectory and pose information, and then feed this information into other models that specialize in behavioral detection and categorization. Deep learning-based software VAME ( Luxem et al, 2022a ) and OpenLabCluster tackle these problems by utilizing previously extracted features ( Li et al, 2022 ). These systems have proven to work well even when the annotated dataset is small.…”

Section: Algorithmsmentioning

confidence: 99%

“…The use of this technology is not restricted to rodent experiments. It has been tested with primates ( Marks et al, 2022 ), broiler chickens ( Fang et al, 2021 ), zebra fish ( Li et al, 2022 ), and fruit flies ( Mathis et al, 2018 ) as well as human eye movements ( Dalmaijer et al, 2014 ).…”

Section: Introductionmentioning

confidence: 99%

Open-source software for automated rodent behavioral analysis

Isik

Ünal

2023

Front. Neurosci.

View full text Add to dashboard Cite

Rodent behavioral analysis is a major specialization in experimental psychology and behavioral neuroscience. Rodents display a wide range of species-specific behaviors, not only in their natural habitats but also under behavioral testing in controlled laboratory conditions. Detecting and categorizing these different kinds of behavior in a consistent way is a challenging task. Observing and analyzing rodent behaviors manually limits the reproducibility and replicability of the analyses due to potentially low inter-rater reliability. The advancement and accessibility of object tracking and pose estimation technologies led to several open-source artificial intelligence (AI) tools that utilize various algorithms for rodent behavioral analysis. These software provide high consistency compared to manual methods, and offer more flexibility than commercial systems by allowing custom-purpose modifications for specific research needs. Open-source software reviewed in this paper offer automated or semi-automated methods for detecting and categorizing rodent behaviors by using hand-coded heuristics, machine learning, or neural networks. The underlying algorithms show key differences in their internal dynamics, interfaces, user-friendliness, and the variety of their outputs. This work reviews the algorithms, capability, functionality, features and software properties of open-source behavioral analysis tools, and discusses how this emergent technology facilitates behavioral quantification in rodent research.

show abstract

“…This section provides a high-level overview of animal behavior classification frameworks for small laboratory animals. We outline a general taxonomy that organizes methods as supervised or unsupervised at a coarse level, and with varying degrees of supervision Taxonomy for Animal Behavior Classification Supervised Classification Hand-crafted Features, Behavior Labels [137], [33], [72], [20], [47], [79], [60], [101], [35], [45] Behavior Labels [91], [160], [63] Hand-crafted Features, Pose and Behavior Labels, PE [139], [145], [146], [4], [94], [121] Pose and Behavior Labels, PE [179] Optical Flow, Hand-crafted Features, Behavior Labels [161] Residual Learning, Optical Flow, Behavior Labels [14] Residual Learning, Pose and Behavior Labels, PE [178] Residual Learning, Optical Flow, Behavior Labels [105] Unsupervised Classification Hand-crafted Features, Pose Labels, PE [61] Fully Unsupervised [144], [11], [172], [8], [16], [73] Fig. 8.…”

Section: Taxonomy For Animal Behavior Classificationmentioning

confidence: 99%

“…-OpenLabCluster [94] contributed unsupervised clustering (Section 2) of pose features using a deep recurrent encoderdecoder architecture and an active learning approach (Section 2). In the latter approach, at each iteration, one sample is labeled, and the cluster map is refined.…”

Section: Taxonomy For Animal Behavior Classificationmentioning

confidence: 99%

CNN-Based Action Recognition and Pose Estimation for Classifying Animal Behavior from Videos: A Survey

Perez¹,

Toler‐Franklin²

2023

Preprint

View full text Add to dashboard Cite

Classifying the behavior of humans or animals from videos is important in biomedical fields for understanding brain function and response to stimuli. Action recognition, classifying activities performed by one or more subjects in a trimmed video, forms the basis of many of these techniques. Deep learning models for human action recognition have progressed significantly over the last decade. Recently, there is an increased interest in research that incorporates deep learning-based action recognition for animal behavior classification. However, human action recognition methods are more developed. This survey presents an overview of human action recognition and pose estimation methods that are based on convolutional neural network (CNN) architectures and have been adapted for animal behavior classification in neuroscience. Pose estimation, estimating joint positions from an image frame, is included because it is often applied before classifying animal behavior. First, we provide foundational information on algorithms that learn spatiotemporal features through 2D, two-stream, and 3D CNNs. We explore motivating factors that determine optimizers, loss functions and training procedures, and compare their performance on benchmark datasets. Next, we review animal behavior frameworks that use or build upon these methods, organized by the level of supervision they require. Our discussion is uniquely focused on the technical evolution of the underlying CNN models and their architectural adaptations (which we illustrate), rather than their usability in a neuroscience lab. We conclude by discussing open research problems, and possible research directions. Our survey is designed to be a resource for researchers developing fully unsupervised animal behavior classification systems of which there are only a few examples in the literature.CCS Concepts: • Computing methodologies → Neural networks.

show abstract

OpenLabCluster: Active Learning Based Clustering and Classification of Animal Behaviors in Videos Based on Automatically Extracted Kinematic Body Keypoints

Cited by 5 publications

References 70 publications

ARBUR, a machine learning-based analysis system for relating behaviors and ultrasonic vocalizations of rats

ARBUR, a machine learning-based analysis system for relating behaviors and ultrasonic vocalizations of rats

Open-source software for automated rodent behavioral analysis

CNN-Based Action Recognition and Pose Estimation for Classifying Animal Behavior from Videos: A Survey

Contact Info

Product

Resources

About