Deep learning with self-supervision and uncertainty regularization to count fish in underwater images

Abstract

Effective conservation actions require effective population monitoring. However, accurately counting animals in the wild to inform conservation decision-making is difficult. Monitoring populations through image sampling has made data collection cheaper, wide-reaching and less intrusive but created a need to process and analyse this data efficiently. Counting animals from such data is challenging, particularly when densely packed in noisy images. Attempting this manually is slow and expensive, while traditional computer vision methods are limited in their generalisability. Deep learning is the state-of-the-art method for many computer vision tasks, but it has yet to be properly explored to count animals. To this end, we employ deep learning, with a density-based regression approach, to count fish in low-resolution sonar images. We introduce a large dataset of sonar videos, deployed to record wild Lebranche mullet schools (Mugil liza), with a subset of 500 labelled images. We utilise abundant unlabelled data in a self-supervised task to improve the supervised counting task. For the first time in this context, by introducing uncertainty quantification, we improve model training and provide an accompanying measure of prediction uncertainty for more informed biological decision-making. Finally, we demonstrate the generalisability of our proposed counting framework through testing it on a recent benchmark dataset of high-resolution annotated underwater images from varying habitats (DeepFish). From experiments on both contrasting datasets, we demonstrate our network outperforms the few other deep learning models implemented for solving this task. By providing an open-source framework along with training data, our study puts forth an efficient deep learning template for crowd counting aquatic animals thereby contributing effective methods to assess natural populations from the ever-increasing visual data.

Computer vision and deep learning for fish classification in underwater habitats: A survey

“…These tasks are mainly designed to extract knowledge from underwater videos and images. Despite the recent use of CNNs for various visual analysis tasks such as segmentation (Alshdaifat et al, 2020;Garcia et al, 2020;Islam et al, 2020;Zhang et al, 2022), localization (Jalal et al, 2020;Knausgård et al, 2021;Su et al, 2020) and counting (Ditria et al, 2021;Schneider & Zhuang, 2020;Tarling et al, 2021), the most common and the widest studied CV task in underwater fish habitat monitoring has been classification. Therefore, in this paper, we focus mainly on classification of underwater fish images.…”

“…A well-known and effective method for improving the generalizability of a DL model is to use regularization (Kukacˇka et al, 2017). Some of the regularization methods applied to fish and marine habitat monitoring domains include transfer learning (Zurowietz & Nattkemper, 2020), batch normalization (Islam et al, 2020), dropout (Iqbal et al, 2021) and using a regularization term (Tarling et al, 2021).…”

Applications of Deep Learning in Fish Habitat Monitoring: A Tutorial and Survey

“…Although the underwater fish counting is limited in the literature, several previous works have advanced the field in this area. For instance, Tarling et al (Tarling et al, 2021) created a novel dataset of sonar video footage of mullet fish labelled manually with point annotations and developed a density-based DL model to count fish from sonar images. They counted fish by using a regression method (Xue et al, 2016) and achieved a MAE of 0.30%.…”

“…The most popular methods of regularisation are L1 and L2. For example, Tarling et al (Tarling et al, 2021) showed that incorporating uncertainty regularisation improves performance of their multi-task network with ResNet-50 (He et al, 2015) backend to count fish in underwater images. (Islam et al, 2020) proposed an optional residual skip block consisting of three convolutional layers with batch normalisation and ReLU non-linearity after each convolutional layer to perform effective semantic segmentation of underwater imagery.…”

A real-time passive acoustic monitoring system to detect Yangtze finless porpoise clicks in Ganjiang River, China

“…Continuous monitoring of the distribution and dynamic changes of endangered species populations is important when planning conservation measures and policies. With the advancements in methodology and technologies, new ways of monitoring animals have been developed as manpower-saving alternatives to human visual observation, such as using drones and cameras to monitor target animals and identify them through image detection algorithms (Atanbori et al, 2015;Li et al, 2021;Tarling et al, 2022). However, for underwater vocalizing animals like porpoises, with some species (such as harbor porpoises, Yangtze finless porpoises) critically endangered (Gallus et al, 2012;Huang et al, 2019), the application of camera technology is limited due to the turbidity and poor visibility of the coastal and riverside water bodies where they live.…”