In situ three‐dimensional video tracking of tagged individuals within site‐attached social groups of coral‐reef fish

Abstract

Tracking the movement of all individual group members in their natural environment remains a challenging task. Using advances in computer vision and Deep Learning, we developed and tested a semi-automated in situ tracking system to reconstruct simultaneous three-dimensional trajectories of marked individuals in social groups of a coral-reef fish. Our system has a temporal resolution of 10s of milliseconds, allowing for multiple 30-min tracking sessions that have been repeated over weeks to months. We present the technique and illustrate its application for Dascyllus marginatus, a planktivorous damselfish that lives in social groups associated with branching corals. Our technique identified all individuals 85-100% of the time, with a mean spatial error of $ 1.3 cm. It provides a cost-effective semi-automated tool for in situ research on movements and foraging of individuals within small site-attached groups of animals in their natural environment.

Big-data approaches lead to an increased understanding of the ecology of animal movement

“…However, they rely on expensive and highly specialized radio transceivers, have limited ability to identify species or individuals, and are usually limited to local or regional scales. Computer-vision algorithms based on modern machine learning approaches, such as convolutional neural networks, can be applied to noninvasively (i.e., without trapping and tagging) track wild birds (47) and fish (49,50,51) in their natural habitats at very high spatiotemporal resolution (e.g., dt = 0.03 s). However, camera tracking in the wild is typically limited to short ranges, an individual's identity cannot be maintained across videos without natural or artificial marking, tracking multiple individuals is still computationally demanding and time-consuming, and the tracking period is usually short (often ≤30 min) or intermittent.…”

Herbivorous fish feeding dynamics and energy expenditure on a coral reef: Insights from stereo‐video and AI‐driven 3D tracking

“…We expect that expanding the iNaturalist dataset with more varied images will enhance our system's ability to accurately classify these species. Our system, capable of tracking and classifying multiple objects, marks a significant advancement over previous studies lacking species identification (Engel et al, 2021;Francisco et al, 2020). Inference of EE in aquatic organisms remains a challenging task and metabolic studies conducted in respiratory chambers -if available at all -seldom capture complex activity patterns observed in the field (Treberg et al, 2016).…”

“…Even further, with the rising application of RUV combined with advanced AI‐driven object recognition and tracking capabilities (Dell et al., 2014 ; Kays et al., 2015 ), our capacity to study animal behavior has improved considerably. Particularly in aquatic environments, remote underwater stereo‐video (RUSV) in combination with AI can meticulously track and analyze the 3D movements of foraging animals (Engel et al., 2021 ; Francisco et al., 2020 ). This innovative approach allows for a broader exploration of animal behavior, providing unprecedented insights into foraging strategies, feeding habits, and energy budgeting (Nathan et al., 2022 ).…”

Analysis of Larval Fish Feeding Behavior under Naturalistic Conditions

“…Only few attempts have been made to automate the analysis of fish behavior from videos under naturalistic conditions. For example, [30] used tracking-by-detection obtain the 3D trajectories of free-swimming reef fish in situ using relatively long video sequences. However, their system was only semi-automatic as all their trajectories were corrected by means of human in the loop.…”