A low-cost, open-source framework for tracking and behavioural analysis of animals in aquatic ecosystems

Abstract

Although methods for tracking animals underwater exist, they frequently involve costly infrastructure investment, or capture and manipulation of animals to affix or implant tags. These practical concerns limit the taxonomic coverage of aquatic movement ecology studies and implementation in areas where high infrastructure investment is impossible. Here we present a method based on deep-learning and structure-from-motion, with which we can accurately determine the 3D location of animals, the structure of the environment in which they are moving. Further behavioural decomposition of the body position and contour of animals subsequently allow quantifying the behavioural states of each interacting animal. This approach can be used with minimal infrastructure and without confining animals to to a fixed area, or capturing and interfering with them in any way. With this approach, we are able to track single individuals (Conger Eel, Conger oceanus), small heterospecific groups (Mullus surmuletus, Diplodus sp.), and schools of animals (Tanganyikan cichlids Lamprologus callipterus) in freshwater and marine systems, and in habitats ranging in environmental complexity. Positional information was highly accurate, with errors as low as 1.67% of body length. Tracking data was embedded in 3D environmental models that could be used to examine collective decision making, obstacle avoidance, and visual connectivity of groups. By analyzing body contour and position, we were also able to use unsupervised classification to quantify the kinematic behavioural states of each animal. The proposed framework allows us to understand animal behaviour in aquatic systems at an unprecedented resolution and a fraction of the cost of established methodologies, with minimal domain expertise at the data acquisition or analysis phase required. Implementing this method, research can be conducted in a wide range of field contexts to collect laboratory standard data, vastly expanding both the taxonomic and environmental coverage of quantitative animal movement analysis with a low-cost, open-source solution. 1/151 Understanding the movement and behaviour of animals in their natural habitats is the 2 ultimate goal of behavioural and movement ecology. By situating our studies in the 3 natural world, we have the potential to uncover the natural processes of selection acting 4 on the behaviour in natural populations, in a manner that cannot be achieved through 5 lab studies alone. The ongoing advance of animal tracking and biologging has the 6 potential to revolutionize not only the scale of data collected from wild systems, but 7 also the types of questions that can subsequently be answered. Incorporating 8 geographical data has already given insights, for example, into the homing behaviour of 9 reef fish, migratory patterns of birds, or the breeding site specificity of sea 10 turtles [7,17,46]. Great advances in systems biology have further been made through 11 the study of movement ecology, understanding migratory patterns of birds traversing 12 their phys...

DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning

“…When combined with unmanned aerial vehicles (UAVs; Schiffman, 2014) or other field-based imaging (Francisco et al, 2019), applying these methods to the study of individuals and groups in the wild can provide high-resolution behavioral data that goes beyond the capabilities of current GPS and accelerometry-based technologies (Nagy et al, 2010; Nagy et al, 2013; Kays et al, 2015; Strandburg-Peshkin et al, 2015; Strandburg-Peshkin et al, 2017; Flack et al, 2018)—especially for species that are impractical to study with tags or collars. Additionally, by applying these methods in conjunction with 3-D habitat reconstruction—using techniques from photogrammetry (Strandburg-Peshkin et al, 2017; Francisco et al, 2019)—field-based studies can begin to integrate fine-scale behavioral measurements with the full 3-D environment in which the behavior evolved. Future advances will likely allow for the calibration and synchronizaton of imaging devices across multiple UAVs (e.g., Price et al, 2018; Saini et al, 2019).…”

“…However, because animals exhibit different behaviors depending on their surroundings (Strandburg-Peshkin et al, 2017; Francisco et al, 2019; Akhund-Zade et al, 2019), laboratory environments are often less than ideal for studying many natural behaviors. Most conventional computer vision methods are also limited in their ability to accurately track groups of individuals over time, but nearly all animals are social at some point in their life and exhibit specialized behaviors when in the presence of conspecifics (Strandburg-Peshkin et al, 2013; Rosenthal et al, 2015; Jolles et al, 2017; Klibaite et al, 2017; Klibaite and Shaevitz, 2019; Francisco et al, 2019; Versace et al, 2019). These methods also commonly track only the animal’s center of mass, which reduces the behavioral output of an individual to a two-dimensional or three-dimensional particle-like trajectory.…”

TRex, a fast multi-animal tracking system with markerless identification, and 2D estimation of posture and visual fields

“…Several technologies are available to separate the foreground from the background (segmentation). Various machine learning algorithms are frequently used to great effect, even for the most complex environments ( Hughey et al 2018, Robie et al 2017, Francisco et al 2019 ). These more advanced approaches are typically beneficial for the analysis of field-data or organisms that are very hard to see in video (e.g.…”

Behavioral traits that define social dominance are the same that reduce social influence in a consensus task

“…We trained an implementation of a Mask and Region based Convolution Neural Network (Mask R-CNN) on a subset of manually labeled images to accurately detect and segment individual fish in the videos, resulting in pixel masks for each video frame and individual respectively (32,33). The masks were then skeletonized using morphological image transformations, allowing to estimate fish spine poses as seven equidistantly spaced points along the midline of each mask's long axis.…”

“…The first and second spine points represent head position and orientation, and were used to automatically reconstruct continuous fish trajectories using a simple, distance-based identity assignment approach. Accuracy and high detection frequency were visually verified with a Python-based GUI developed within the lab, that was also used to manually correct false identity assignments and losses (32).…”