Social Behavior Atlas: A computational framework for tracking and mapping 3D close interactions of free-moving animals

Abstract

The study of social behaviors in animals is essential for understanding their survival and reproductive strategies. However, accurately tracking and analyzing the social interactions of free-moving animals has remained a challenge. Existing multi-animal pose estimation techniques suffer from drawbacks such as the need for extensive manual annotation and difficulty in discriminating between similar-looking animals in close social interactions. In this paper, we present the Social Behavior Atlas (SBeA), a novel computational framework that solves these challenges by employing a deep learning-based video instance segmentation model, 3D pose reconstruction, and unsupervised dynamic behavioral clustering. SBeA framework also involves a multi-camera setup to prevent occlusion, and a novel approach to identify individual animals in close social interactions. We demonstrate the effectiveness of SBeA in tracking and mapping the 3D close interactions of free-moving animals using the example of genetic mutant mice, birds, and dogs. Our results show that SBeA is capable of identifying subtle social interaction abnormalities, and the models and frameworks developed can be applied to a wide range of animal species. SBeA is a powerful tool for researchers in the fields of neuroscience and ecology to study animal social behaviors with a high degree of accuracy and reliability.

Anti-drift pose tracker (ADPT): A transformer-based network for robust animal pose estimation cross-species

“…This structure design makes ADPT show significantly fewer tracking drifts than DeepLabCut and SLEAP Mathis et al (2018 ); Pereira et al (2022 ). The effect of anti-drift of ADPT is universally validated in the public datasets and our customized datasets including Drosophilas, mice, and macaques, which demonstrates that ADPT is robust in broad application scenarios cross-species Pereira et al (2019 ); Bala et al (2020 ); Han et al (2023a ). ADPT also achieves robust pose estimation and identity recognition of free-interactive mice combined with a mix-up dataset generation strategy.…”

“…In addition, the well-annotated animal pose datasets are not abundant enough to cover various experiment settings. Experimenters always need to make customized datasets for their specific applications Han et al (2023a ). Therefore, the application of the Transformer to reduce tracking drift in the animal pose estimation task still needs an elaborate design of ANN structures.…”

“…ADPT also achieves robust pose estimation and identity recognition of free-interactive mice combined with a mix-up dataset generation strategy. The results of markerless identity recognition show that the feature extraction of ADPT is reliable enough to cover both multi-animal pose estimation and identity recognition tasks, which are more difficult than the single-animal pose estimation task Agezo and Berman (2022 ); Lauer et al (2022 ); Han et al (2023a ). It reduces the computational time cost and increases the throughput of behavioral data processing because ADPT does not need a multi-stage neural network such as SIPEC Marks et al (2022 ) or Social Behavior Atlas Han et al (2023a ).…”

“…The results of markerless identity recognition show that the feature extraction of ADPT is reliable enough to cover both multi-animal pose estimation and identity recognition tasks, which are more difficult than the single-animal pose estimation task Agezo and Berman (2022 ); Lauer et al (2022 ); Han et al (2023a ). It reduces the computational time cost and increases the throughput of behavioral data processing because ADPT does not need a multi-stage neural network such as SIPEC Marks et al (2022 ) or Social Behavior Atlas Han et al (2023a ). Together, ADPT would be an accessible tool to reduce the pose tracking shift across species from the upstream of behavior analysis.…”

MiceVAPORDot: A novel automated approach for high-throughput behavioral characterization during E-cigarette exposure in mice

“…More cameras in different directions can be added to MiceVAPORDot for the 3D reconstruction of each mouse (Han et al, 2022). Additionally, the behavioral phenotype profile can be a more comprehensive description of the effects of e-cigarettes (Huang et al, 2021; Liu et al, 2021), the accurate tracking of animals requires massive dataset annotations for training the deep learning model which can be simplified by generative data annotation methods to increase the speed of data processing (Han et al, 2023).…”