Real-time, low-latency closed-loop feedback using markerless posture tracking

Abstract

The ability to control a behavioral task or stimulate neural activity based on animal behavior in real-time is an important tool for experimental neuroscientists. Ideally, such tools are noninvasive, low-latency, and provide interfaces to trigger external hardware based on posture. Recent advances in pose estimation with deep learning allows researchers to train deep neural networks to accurately quantify a wide variety of animal behaviors. Here, we provide a new <monospace>DeepLabCut-Live!</monospace> package that achieves low-latency real-time pose estimation (within 15 ms, >100 FPS), with an additional forward-prediction module that achieves zero-latency feedback, and a dynamic-cropping mode that allows for higher inference speeds. We also provide three options for using this tool with ease: (1) a stand-alone GUI (called <monospace>DLC-Live! GUI</monospace>), and integration into (2) <monospace>Bonsai,</monospace> and (3) <monospace>AutoPilot</monospace>. Lastly, we benchmarked performance on a wide range of systems so that experimentalists can easily decide what hardware is required for their needs.

Neuroscience Cloud Analysis As a Service

“…First, we hope to build a community of developers who will add more analysis algorithms to N euro CAAS, with an emphasis on subfields of computational analysis that we do not yet support. We also plan to add support for real-time processing (e.g., Giovannucci et al (2017) for calcium imaging, or Schweihoff et al, 2019, Kane et al, 2020 for closed-loop experiments, or Lopes et al (2015) for the coordination of multiple data streams). Second, other tools have brought large-scale distributed computing to neural data analyses (Freeman, 2015, Rocklin, 2015) in ways that conform to more traditional high performance computing ideas of scalability for applications that are less easily parallelized than those presented here.…”

New Open-Source Tools: Using Bonsai for Behavioral Tracking and Closed-Loop Experiments

“…Expanding the basic center of mass tracking to specific body parts, using DeepLabCut or other simple image processing techniques, expands the possibilities for virtual fixation even further, allowing fixing specifically the video around the nose or head of the animal, and then calculating distances between those body parts and other points of interest (or other animals; Kane et al, 2020).…”

“…Crucially, the experimenter is now in control of the interaction and can modify the response characteristics of the closed-loop system to investigate animal behavior, for instance by delaying, suppressing, or amplifying the feedback response parametrically. Using Bonsai, virtual fixation techniques can themselves be used in the design of such closed-loop systems, for example by using the increasing computational capabilities of GPUs for real-time pose estimation (Kane et al, 2020).…”

“…Since its publication in 2015 , Bonsai has been widely used by many labs worldwide not only for tracking animal behavior in different species of rodents, cephalopods, fish, and insects (Dreosti et al, 2015;Walker et al, 2015;Douglass et al, 2017) but also to control and acquire data from multiple streams. Being open-source software, Bonsai is free to use and not proprietary to any one company, thus many users have adopted it to create their specific packages for EEG (Lopes, 2018), Miniscopes (Aharoni and Hoogland, 2019;Guo et al, 2020), Fiber photometry (Carvalho and Lopes, 2019), Open Ephys (Neto et al, 2016) and real-time video analysis with DeepLabCut (Kane et al, 2020). It also encourages good practices for experimental reproducibility by including a built-in package manager and support for portable deployment across rigs.…”

Wide-Angle, Monocular Head Tracking using Passive Markers

“…With the rising prominence of DeepLabCut for markerless tracking of laboratory animal features [9] such multi-camera 3D pose estimation systems based on DeepLabCut have been introduced [8,38]. DeepLabCut-based systems can also be used to track features in real time [39,40,41], although processing speeds are severely limited by image resolution. While these algorithms can run using a CPU alone, their performance is degraded by up to 100 fold [38], rendering them too slow for real-time use.…”