Real-time, low-latency closed-loop feedback using markerless posture tracking

Abstract

The ability to control a behavioral task or stimulate neural activity based on animal behavior in real-time is an important tool for experimental neuroscientists. Ideally, such tools are (1) noninvasive, (2) low-latency, and (3) provide interfaces to trigger external hardware based on posture (i.e., not just objectbased-tracking). Recent advances in pose estimation with deep learning allows researchers to train deep neural networks to accurately quantify a wide variety of animal behaviors. In extending our efforts towards the animal pose estimation toolbox DeepLabCut, here, we provide a new DeepLabCut-Live! package that achieves low-latency real-time pose estimation (within 15 ms, at >100 FPS), with an additional forwardprediction module that achieves zero-latency feedback. We also provide three options for using this tool with ease: a stand-alone GUI (called DLC-Live!GUI), integration into Bonsai and into AutoPilot. Lastly, we benchmarked performance on a wide range of systems so that experimentalists can easily decide what hardware is required for their needs.HighlightsThe DeepLabCut-Live! package is available via pip install deeplabcut-liveThe Bonsai-DLC plugin is availableThe AutoPilot-DLC plugin is availableThe DeepLabCut-Live! GUI package is available via pip install deeplabcut-live-gui

Real-Time Closed-Loop Feedback in Behavioral Time Scales Using DeepLabCut

“…Our current work extends the use of these approaches for real-time marker-less tracking of multiple whiskers. A few recent studies have made forays into real-time tracking with DeepLabCut ( Forys et al, 2020 ; Kane et al, 2020 ). The first study using this approach demonstrated control of mouse behavior by rewarding particular forelimb movements, achieving a real-time tracking latency of 50 ms ( Forys et al, 2020 ).…”

“…The first study using this approach demonstrated control of mouse behavior by rewarding particular forelimb movements, achieving a real-time tracking latency of 50 ms ( Forys et al, 2020 ). The other one, using a different approach, has achieved a latency of 10 ms ( Kane et al, 2020 ). But to achieve a 10-ms latency, Kane and colleagues subsampled each frame and restricted the resolution of video frames to ∼150 pixels in diameter.…”

“…One effective procedure would be the use of more compact DNN models such as MobileNetV2 ( Mathis et al, 2019 ). It could help reduce the load on the GPU and thus could lead to higher efficiency in body-part estimation ( Kane et al, 2020 ). Restricting the region of interest in video frames could also help keep accuracy from degrading while speeding up the estimation process ( Nath et al, 2019 ; Sehara et al, 2019 ).…”

Closed-loop automated reaching apparatus (CLARA) for interrogating motor systems

“…Multiple studies have implemented markerless tracking of forelimb reach, (Guo et al, 2015; Galiñanes et al, 2018), including some that used DeepLabCut frameworks to provide real-time closed-loop stimulation (Kane et al, 2020; Vonstad et al, 2020; Sehara et al, 2021). However, the majority of these systems involve head fixation, which greatly constrains movement and limits the translational relevance.…”

“…However, the majority of these systems involve head fixation, which greatly constrains movement and limits the translational relevance. Further, even in systems that could be used in freely moving contexts (Kane et al, 2020), closed-loop stimulation is limited to positional or positional derivative kinematic thresholds. To our knowledge, CLARA is the first system that can accurately provide close-loop feedback in response to an unconstrained behavior outcome.…”