Deep learning-assisted comparative analysis of animal trajectories with DeepHL

Abstract

A comparative analysis of animal behavior (e.g., male vs. female groups) has been widely used to elucidate behavior specific to one group since pre-Darwinian times. However, big data generated by new sensing technologies, e.g., GPS, makes it difficult for them to contrast group differences manually. This study introduces DeepHL, a deep learning-assisted platform for the comparative analysis of animal movement data, i.e., trajectories. This software uses a deep neural network based on an attention mechanism to automatically detect segments in trajectories that are characteristic of one group. It then highlights these segments in visualized trajectories, enabling biologists to focus on these segments, and helps them reveal the underlying meaning of the highlighted segments to facilitate formulating new hypotheses. We tested the platform on a variety of trajectories of worms, insects, mice, bears, and seabirds across a scale from millimeters to hundreds of kilometers, revealing new movement features of these animals.

Non-Parametric Analysis of Inter-Individual Relations Using an Attention-Based Neural Network

“…They showed that the attention mechanism can locate EEG channels that specifically diagnose seizures. Similarly, Maekawa et al (2020) analyzed movements of worms and mice using the attention mechanism and highlighted their characteristic movements under different experimental conditions. These studies reemphasize that attention-based data analyses target situations in which researchers do not know which aspects/features/portions of the data are most important; this limitation is overcome by implementing a data-driven detection of relevant features.…”

“…Unlike conventional analysis, the proposed approach automatically detects inter-individual relations from raw data using a machine learning technique (Figure 1; cf. Maekawa et al, 2020). For example, location logs of animals collected through bio-logging devices can be analyzed without the need to first determine relevant aspects of the data such as co-appearance within a certain distance.…”

“…It is now considered central technology in today’s natural language processing (NLP). The present study applies this technology to reveal latent relations among animal agents that are reflected in their location data, without prior assumptions about the possible types of relationship structures or noise distributions (Figure 2c; see Yuan and Jia, 2019; Maekawa et al, 2020, for other applications to biological studies). While only the location data are discussed in this study, there are many possible applications of our relation analysis given the flexibility of the neural network.…”

Cross-species Behavior Analysis with Attention-based Domain-adversarial Deep Neural Networks

“…DeepHL, which is our prior work, is a pioneering study on deep learning-assisted animal behavior analysis using attention mechanisms 27 . However, DeepHL focuses only on behavioral data from a single species and cannot be used to conduct cross-species behavior analysis.…”

Deep learning as a tool for ecology and evolution

“…Several open-source toolkits have been developed for this purpose, ranging from species-specific solutions (e.g., DeepFly3D for Drosophila (Günel et al, 2019), OpenMonkeyStudio for macaques (Bala et al, 2020)) to generic frameworks applicable to any species (e.g., LEAP (Pereira et al, 2019(Pereira et al, , 2020, DeepLabCut (Mathis et al, 2018;Nath et al, 2019), DeepPoseKit (Graving et al, 2019)), some of which offer 3-dimensional and/or multiple animals tracking. In addition to pose estimation, deep learning is also being adopted to enhance the performance of established computer vision methods used to track spatial position of animals (e.g., by tag detection (Sixt et al, 2018) or the identification of markers (Gal et al, 2020)), as well as to automatically perform behavioral analysis of spatial trajectories (Maekawa et al, 2020).…”