The desire to understand how the brain generates and patterns behavior has driven rapid methodological innovation in tools to quantify natural animal behavior. While advances in deep learning and computer vision have enabled markerless pose estimation in individual animals, extending these to multiple animals presents unique challenges for studies of social behaviors or animals in their natural environments. Here we present Social LEAP Estimates Animal Poses (SLEAP), a machine learning system for multi-animal pose tracking. This system enables versatile workflows for data labeling, model training and inference on previously unseen data. SLEAP features an accessible graphical user interface, a standardized data model, a reproducible configuration system, over 30 model architectures, two approaches to part grouping and two approaches to identity tracking. We applied SLEAP to seven datasets across flies, bees, mice and gerbils to systematically evaluate each approach and architecture, and we compare it with other existing approaches. SLEAP achieves greater accuracy and speeds of more than 800 frames per second, with latencies of less than 3.5 ms at full 1,024 × 1,024 image resolution. This makes SLEAP usable for real-time applications, which we demonstrate by controlling the behavior of one animal on the basis of the tracking and detection of social interactions with another animal.

The desire to understand how the brain generates and patterns behavior has driven rapid methodological innovation to quantify and model natural animal behavior. This has led to important advances in deep learning-based markerless pose estimation that have been enabled in part by the success of deep learning for computer vision applications. Here we present SLEAP (Social LEAP Estimates Animal Poses), a framework for multi-animal pose tracking via deep learning. This system is capable of simultaneously tracking any number of animals during social interactions and across a variety of experimental conditions. SLEAP implements several complementary approaches for dealing with the problems inherent in moving from single-to multi-animal pose tracking, including configurable neural network architectures, inference techniques, and tracking algorithms, enabling easy specialization and tuning for particular experimental conditions or performance requirements. We report results on multiple datasets of socially interacting animals (flies, bees, and mice) and describe how dataset-specific properties can be leveraged to determine the best configuration of SLEAP models. Using a high accuracy model (<2.8 px error on 95% of points), we were able to track two animals from full size 1024 × 1024 pixel frames at up to 320 FPS. The SLEAP framework comes with a sophisticated graphical user interface, multi-platform support, Colab-based GPU-free training and inference, and complete tutorials available, in addition to the datasets, at sleap.ai.

No abstract

3D eukaryotic genome organization provides the structural basis for gene regulation. In Drosophila melanogaster, genome folding is characterized by somatic homolog pairing, where homologous chromosomes are intimately paired from end to end; however, how homologs identify one another and pair has remained mysterious. Recently, this process has been proposed to be driven by specifically interacting 'buttons' encoded along chromosomes. Here, we turned this hypothesis into a quantitative biophysical model to demonstrate that a button-based mechanism can lead to chromosome-wide pairing. We tested our model using live-imaging measurements of chromosomal loci tagged with the MS2 and PP7 nascent RNA labeling systems. We show solid agreement between model predictions and experiments in the pairing dynamics of individual homologous loci. Our results strongly support a button-based mechanism of somatic homolog pairing in Drosophila and provide a theoretical framework for revealing the molecular identity and regulation of buttons.

The spatial configuration of the eukaryotic genome is organized and dynamic, providing the structural basis for regulated gene expression in living cells. In Drosophila melanogaster, 3D genome organization is characterized by the phenomenon of somatic homolog pairing, where homologous chromosomes are intimately paired from end to end. While this organizational principle has been recognized for over 100 years, the process by which homologs identify one another and pair has remained mysterious. Recently, a model was proposed wherein specifically-interacting “buttons” encoded along the lengths of homologous chromosomes drive somatic homolog pairing. Here, we turn this hypothesis into a precise biophysical model to demonstrate that a button-based mechanism can lead to chromosome-wide pairing. We test our model and constrain its free parameters using live-imaging measurements of chromosomal loci tagged with the MS2 and PP7 nascent RNA labeling systems. Our analysis shows strong agreement between model predictions and experiments in the separation dynamics of tagged homologous loci as they transition from unpaired to paired states, and in the percentage of nuclei that become paired as embryonic development proceeds. In sum, our data strongly support a button-based mechanism of somatic homolog pairing in Drosophila and provide a theoretical framework for revealing the molecular identity and regulation of buttons.