A Markerless 3D Computerized Motion Capture System Incorporating a Skeleton Model for Monkeys

Abstract

In this study, we propose a novel markerless motion capture system (MCS) for monkeys, in which 3D surface images of monkeys were reconstructed by integrating data from four depth cameras, and a skeleton model of the monkey was fitted onto 3D images of monkeys in each frame of the video. To validate the MCS, first, estimated 3D positions of body parts were compared between the 3D MCS-assisted estimation and manual estimation based on visual inspection when a monkey performed a shuttling behavior in which it had to avoid obstacles in various positions. The mean estimation error of the positions of body parts (3–14 cm) and of head rotation (35–43°) between the 3D MCS-assisted and manual estimation were comparable to the errors between two different experimenters performing manual estimation. Furthermore, the MCS could identify specific monkey actions, and there was no false positive nor false negative detection of actions compared with those in manual estimation. Second, to check the reproducibility of MCS-assisted estimation, the same analyses of the above experiments were repeated by a different user. The estimation errors of positions of most body parts between the two experimenters were significantly smaller in the MCS-assisted estimation than in the manual estimation. Third, effects of methamphetamine (MAP) administration on the spontaneous behaviors of four monkeys were analyzed using the MCS. MAP significantly increased head movements, tended to decrease locomotion speed, and had no significant effect on total path length. The results were comparable to previous human clinical data. Furthermore, estimated data following MAP injection (total path length, walking speed, and speed of head rotation) correlated significantly between the two experimenters in the MCS-assisted estimation (r = 0.863 to 0.999). The results suggest that the presented MCS in monkeys is useful in investigating neural mechanisms underlying various psychiatric disorders and developing pharmacological interventions.

Wireless recording from unrestrained monkeys reveals motor goal encoding beyond immediate reach in frontoparietal cortex

“…Its technical realization is far from simple, though, especially with the complex movements primates are capable of. When using wireless technology, a desirable approach would be to let the monkey freely decide on their behavior to obtain neural correlates of most natural behavior (Gilja et al, 2010) while motion capture provides the related movement kinematics (Ballesta et al, 2014;Bansal, Truccolo, Vargas-Irwin, & Donoghue, 2012;Nakamura et al, 2016;Peikon et al, 2009). But even if full-body motion capture would be available, it will remain a major challenge to identify to what extent neural activity relates to sensory input, the currently performed movement or the planning of the next movement in free behavior.…”

“…An alternative are fluorescent or reflective markers directly painted to the skin of the animal (Courtine et al, 2005;Peikon, Fitzsimmons, Lebedev, & Nicolelis, 2009), which also require continuously repeated shaving, or markers that cannot be removed, such as collars (Ballesta, Reymond, Pozzobon, & Duhamel, 2014). A videobased marker-free system using skeleton models was recently reported (Nakamura et al, 2016), however, this or similar systems were not yet reported in a larger, more complex environment with monkeys. We used a commercially available system with only four VGA cameras tracking a permanently dyed part on the animal's fur and the colored cap of the head implant (data not shown).…”

MacaquePose: A novel ‘in the wild’ macaque monkey pose dataset for markerless motion capture

“…The high-resolution spatiotemporal data obtained with the markerless motion capture will also aid in understanding brain dynamics underlying the behavior (Berger et al, 2020). Specific posture and motion are informative for studying animals' emotions and intension (Nakamura et al, 2016), and the motor functions (Berger et al, 2020). Furthermore, the automatic and long-term analyses of naturalistic behavior from a large number of subjects permit new data-driven approaches to find unusual behaviors, personalities and, underlying genetic and neural mechanisms (Vogelstein et al, 2014;De Chaumont et al, 2018).…”

“…Motion capture technologies allow the quantification of animal's pose and motion with a high spatiotemporal resolution enabling the study of the relationship between various brain functions and behaviors (Vargas-Irwin et al, 2008;Nagasaka et al, 2011;Mathis et al, 2020). However, attaching the physical markers for the motion capture is often not practical for animal studies, as the markers themselves disturb/change the subject's behavior (Nakamura et al, 2016;Mathis et al, 2018;Berger et al, 2020). Thanks to recent advances in machine vision using deep learning, the video-based markerless motion capture has been developed to a level permitting practical use (Mathis et al, 2020), in which an artificial neural network predicts the location of body parts in a video without the requirement for physical markers, and enabled successful behavioral studies in rodents (e.g., Dooley et al, 2020;Cregg et al, 2020;Mathis et al, 2020).…”

Monkey Features Location Identification Using Convolutional Neural Networks

“…However, the markers can affect the animals natural behavior. For this reason, lowcost marker-less motion capture systems for animal tracking are developed and introduced in [2] and [3] using depth sensors. Recent studies also utilized convolutional neural networks to do marker-less 2D animal pose estimation such as [5]; however, they have focusd more in laboratory-controlled setups and their results are limited to the fruit fly and mouse.…”