Deep learning-based system for real-time behavior recognition and closed-loop control of behavioral mazes using depth sensing

Abstract

Robust quantification of animal behavior is fundamental in experimental neuroscience research. Systems providing automated behavioral assessment are an important alternative to manual measurements avoiding problems such as human bias, low reproducibility and high cost. Integrating these tools with closed-loop control systems creates conditions to correlate environment and behavioral expressions effectively, and ultimately explain the neural foundations of behavior. We present an integrated solution for automated behavioral analysis of rodents using deep learning networks on video streams acquired from a depth-sensing camera. The use of depth sensors has notable advantages: tracking/classification performance is improved and independent of animals' coat color, and videos can be recorded in dark conditions without affecting animals' natural behavior. Convolutional and recurrent layers were combined in deep network architectures, and both spatial and temporal representations were successfully learned for a 4-classes behavior classification task (standstill, walking, rearing and grooming). Integration with Arduino microcontrollers creates an easy-to-use control platform providing low-latency feedback signals based on the deep learning automatic classification of animal behavior. The complete system, combining depth-sensor camera, computer, and Arduino microcontroller, allows simple mapping of input-output control signals using the animal's current behavior and position. For example, a feeder can be controlled not by pressing a lever but by the animal behavior itself. An integrated graphical user interface completes a user-friendly and cost-effective solution for animal tracking and behavior classification. This open-software/open-hardware platform can boost the development of customized protocols for automated behavioral research, and support ever more sophisticated, reliable and reproducible behavioral neuroscience experiments.

ARBUR, a machine learning-based analysis system for relating behaviors and ultrasonic vocalizations of rats

“…The confusion matrix shows that AR-BUR:Behavior can discriminate the easy-to-confuse social behaviors with high F1-scores, such as PIN (pinning) and POU (pouncing) (Figure 3c). In contrast to studies based on end-to-end architectures Bohnslav et al ( 2021 ); Marks et al ( 2022 ); Gerós et al ( 2022 ), ARBUR:Behavior is capable of discriminating specific social behaviors (PIN, POU, and SNC (social nose contact) and can classify a higher number of behavior categories while achieving comparable accuracy.…”

“…3c). In contrast to studies based on end-to-end architectures [19, 31, 33], ARBUR:Behavior is capable of discriminating specific social behaviors (PIN, POU, and SNC (social nose contact) and can classify a higher number of behavior categories while achieving comparable accuracy.…”

ARBUR, a machine learning-based analysis system for relating behaviors and ultrasonic vocalizations of rats

“…The confusion matrix shows that ARBUR:Behavior can discriminate the easy-to-confuse social behaviors with high F1-scores, such as PIN (pinning) and POU (pouncing) ( Figure 3 C). In contrast to studies based on end-to-end architectures, 19 , 32 , 33 ARBUR:Behavior is capable of discriminating specific social behaviors (PIN, POU, and SNC (social nose contact)) and can classify a higher number of behavior categories while achieving comparable accuracy.

Figure 3 ARBUR simultaneously achieves accurate behavior detection and sound source localization in three dimensions (A) Examples of the ARBUR outputs for eight behaviors: left and right camera view, behavior type (top left of each panel), spectrograms of USVs with cluster (inset, top left), and duration (inset, bottom right) and location of the vocal rat in the two camera views (red circle).

…”

Open-source software for automated rodent behavioral analysis

“…It removes the background from the videos, which allows models to learn features better from relevant signals by eliminating noise, and then extracts animation and positional changes of the animal. In addition to using pre-trained model weights, DeepEthogram ( Bohnslav et al, 2021 ) and DeepCaT-z ( Gerós et al, 2022 ), respectively, incorporate optic flow and depth information into the video frames to overcome the need for large datasets. TREBA ( Sun et al, 2021 ) proposes a unique approach that utilizes both expert knowledge and neural networks.…”

“…This separation of functionality allows researchers to experiment with a combination of tracking, pose estimation, behavior analysis software, and find the most suitable combination for their specific needs. Certain software such as AlphaTracker ( Chen et al, 2020 ), MARS ( Segalin et al, 2021 ), SIPEC ( Marks et al, 2022 ), LabGym ( Hu et al, 2022 ), DeepCaT-z ( Gerós et al, 2022 ), and DeepEthogram ( Bohnslav et al, 2021 ) offer a combination of these solutions, enabling a comprehensive behavioral analysis within the same system ( Table 2 ). This is especially useful for researchers who do not have the necessary technical skills to merge the inputs and outputs of different tools.…”