A system for tracking whisker kinematics and whisker shape in three dimensions

Abstract

25Quantification of behaviour is essential for systems neuroscience. Since the whisker system is a major model system for 26 investigating the neural basis of behaviour, it is important to have methods for measuring whisker movements from 27 behaving animals. Here, we developed a high-speed imaging system that measures whisker movements simultaneously 28 from two vantage points. We developed an algorithm that uses the 'stereo' video data to track multiple whiskers by 29 fitting 3D curves to the basal section of each target whisker. By using temporal information to constrain the fits, the 30 algorithm is able to track multiple whiskers in parallel with low error rate. We used the output of the tracker to produce a 31 3D description of each tracked whisker, including its 3D orientation and 3D shape, as well as bending-related mechanical 32 force. In conclusion, we present an automatic system to track whiskers in 3D from high-speed video, creating the 33 opportunity for comprehensive 3D analysis of sensorimotor behaviour and its neural basis. 34 Author summary 35The great ethologist Niko Tinbergen described a crucial challenge in biology to measure the "total movements made by 36 the intact animal". Advances in high-speed video and machine analysis of such data have made it possible to make 37 profound advances. Here, we target the whisker system. The whisker system is a major experimental model in 38 neurobiology and, since the whiskers are readily imageable, the system is ideally suited to machine vision. Rats and mice 39 explore their environment by sweeping their whiskers to and fro. It is important to measure whisker movements in 3D, 40 since whiskers move in 3D and since the mechanical forces that act on them are 3D. However, the problem of 41 automatically tracking whiskers in 3D from video has generally been regarded as prohibitively difficult. Our innovation 42 here is to extract 3D information about whiskers using a two-camera, high-speed imaging system and to develop 43 computational methods to infer 3D whisker state from the imaging data. Our hope is that this study will facilitate 44 comprehensive, 3D analysis of whisker behaviour and, more generally, contribute new insight into brain mechanisms of 45 perception and behaviour. 46 47 48 50 Substantial progress towards the long-standing ambition of measuring "total movements made by the intact animal" (1) is 51 coming from the application of powerful machine vision methods to video recordings of behaving animals (2). Since the 52 whisker system is a major experimental model in neuroscience and since the whiskers are readily imageable (3,4), the 53 whisker system is ideally suited to this endeavour. Tracking the whiskers of mice/rats has already deepened our 54 understanding of active sensation and refined our capacity to relate behaviour to neural mechanisms (5-10) Our aim here 55 was to develop a method to track whisker movements and whisker shape in 3D in behaving mice at millisecond temporal 56 resolution. 57Whisker movement is 3D. During each whisking...

Ecomorphology reveals Euler spiral of mammalian whiskers

“…Other studies have approximated whiskers as quadratic curves (Quist & Hartmann, 2012; Towal et al, 2011), which cannot replicate inflections in curvature, or otherwise used cubic splines to capture whisker shape (Bagdasarian et al, 2013; Belli, Bresee, Graff, & Hartmann, 2018), but these are challenging to compare with one another. It is also possible to compare whisker outlines fitted with Elliptic Fourier harmonic coefficients (Ginter et al, 2012) or Bezier curves (Campagner, Evans, Loft, & Petersen, 2018; Hewitt et al, 2018; Petersen, Colins, Evans, Campagner, & Loft, 2020); although these are good visual representations of whisker shape, they do not provide a clear and succinct equation, which is useful for developing mechanical models. Therefore, as previously observed in rats (Starostin et al, 2020), we propose that a two parameter, linear curvature function (Equation ) provides a good approximation for whisker curvature.…”

A High-Dimensional Quantification of Mouse Defensive Behaviors Reveals Enhanced Diversity and Stimulus Specificity

“…Such reconstructions have been first developed for constrained situations (e.g., treadmill walk) and by applying physical markers to detect body landmarks [ 25 ]. More recently, machine learning [ 26 , 27 , 28 ] and deep learning [ 29 , 30 , 31 ] have allowed to obviate for the need to use physical markers. Alternative approaches have also been taken by using depth cameras [ 32 ] or by combining traditional video with head-mounted sensors to measure head movements [ 33 ] and even eye movements and pupil constriction [ 34 ].…”

Real-Time Closed-Loop Feedback in Behavioral Time Scales Using DeepLabCut

“…In the last 20 years, multiple research groups have reported increasingly sophisticated, easy-to-use tools for automated annotation of video data of animal behavior (Knutsen et al, 2005;Voigts et al, 2008;Perkon et al, 2011;Clack et al, 2012;Ohayon et al, 2013;Giovannucci et al, 2018;Dominiak et al, 2019;Vanzella et al, 2019;Betting et al, 2020;Petersen et al, 2020). One natural extension of this ability has been to apply these algorithms for on-line, closed-loop paradigms, where changes in behavior of the animal are detected as rapidly as possible, and the behavior is used to modify or manipulate the brain, the virtual environment or the context of behavior.…”

“…Traditionally, video data have been analyzed manually. More recently, various algorithms have been developed for automating movement detection (Knutsen et al, 2005;Voigts et al, 2008;Perkon et al, 2011;Clack et al, 2012;Ohayon et al, 2013;Giovannucci et al, 2018;Dominiak et al, 2019;Vanzella et al, 2019;Betting et al, 2020;Petersen et al, 2020). With the development of DeepLabCut, a marker-less pose-estimation toolkit based on deep learning , computer vision approaches are being used for monitoring poses of animals and for tracking the movement of virtually any part of the body.…”