Computer Vision Reveals Fish Behaviour Through Structural Equation Modelling of Movement Patterns

Abstract

Background: Studying and quantifying fish behaviour is important to understand how fish interact with their environments. Yet much of fish behaviour in aquatic ecosystems remains hard to observe and time-consuming to manually document. Automated tracking through computer vision techniques can provide fine-scale movement data of many individuals across spatial and temporal scales. When used alongside statistical methodologies such as structural equation models (SEMs), these data can be used to infer underlying latent behavioural events of an animal without direct observations. However, interpreting the main modes of fish behaviour when presented with large datasets remains a challenge. Methods: We developed a cross-validation routine to evaluate methods for accurately predicting behaviour from a large field dataset. We created SEM behavioural models using different combinations of temporal buffers in training data and evaluated them against a fixed evaluation dataset for three common fish species (Moses perch, southern herring and yellowfin bream). We then used the SEM model with the lowest error for each species to translate movement data into ecologically relevant behavioural events. Results: The temporal buffer generating the SEM models with the best predictive capacity varied among species, from 1 min for Moses perch and southern herring, to 45 min for yellowfin bream. Models were able to accurately predict behavioural events such as foraging (slow, sinuous movement near the substrate) and fine-scale diurnal migration (fast, directional movement near the surface).Conclusions: We developed a framework to optimise temporal partitioning in computer vision tracking data using SEMs. We demonstrate how data can be translated into ecologically relevant behavioural events such as foraging and fine-scale migrations. This is a step towards bridging the gap between development and application of new technologies to monitor behaviour of animals in aquatic ecosystems.

Fish surveys on the move: Adapting automated fish detection and classification frameworks for videos on a remotely operated vehicle in shallow marine waters

“…Even for the two better performing frameworks, accuracy for counts per frame was below 80% (73.8% for Detectron, 69.3% for YOLOv5, as well as for MaxN (73% for Detectron, 65.3% for YOLOv5). Object detection performance on stationary cameras for this species has had a wide range of outcomes in recent studies, namely: 91% F1 value in a single species model (Lopez-Marcano et al, 2021), but also 75.0% in a three species model (Lopez-Marcano et al, 2022). For bream, we are unable to clearly state whether automation algorithms are performing differently on mobile than on stationary cameras.…”

“…CNN models in recent multispecies analyses of videos from underwater stationary cameras have produced overall performances (as % of object detection correct, similar to F1) of: 86.9% average on 18 fish species (Villon et al, 2018), and 78.0% on 20 species (with values for individual species ranging from 63 -99%, Villon et al, 2021). Single species CNN models for stationary cameras in local waters near the current survey location have produced a range of F1 values in object detection analyses: 87.6% and 92.3% over seagrass habitat (Ditria et al, 2020a;Ditria et al, 2020b); 83.0% and 90.6% over reef (Ditria et al, 2020b;Lopez-Marcano et al, 2022), albeit with much lower values where training did not include the habitat over which test videos were filmed (e.g. 58.4% and 73.3%, Ditria et al, 2020b).…”

Estimating precision and accuracy of automated video post-processing: A step towards implementation of AI/ML for optics-based fish sampling

“…Rugose habitats of reefs may lead to larger numbers of false negatives/positives in fish detections due to cryptic behavior and/or coloring and mottling that resembles complex habitat (e.g., lionfish). Fish species of different size classes and with different swimming or schooling behaviors may be harder to detect or classify than others (Lopez-Marcano et al, 2022), especially at variable distances from a camera at a fixed position (mobile cameras face their own challenges). Regardless of the source of error, the main challenge is that the annotation phase of postprocessing is likely impacted by detection and identification differences arising from variable environmental conditions in which video is collected, and therefore great care has to be taken to ensure that time-series remain stable relative to changes made in post-processing methods.…”