FIN-PRINT a fully-automated multi-stage deep-learning-based framework for the individual recognition of killer whales

Bergler, Christian; Gebhard, Alexander; Towers, Jared R.; Butyrev, Leonid; Sutton, Gary J.; Shaw, Tasli J. H.; Maier, Andreas; Nöth, Elmar

doi:10.1038/s41598-021-02506-6

Cited by 11 publications

(5 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While this model has been compared to the other 39,284 models in the competition, it has not been explicitly compared to cetacean photo–id models in the literature (e.g. Bergler et al, 2021; Maglietta et al, 2020, 2023; Thompson et al, 2021; Weideman et al, 2017). A full comparison (sensu Tyson Moore et al, 2022), however, would be challenging because we are unaware of any model that could be reasonably used to predict the identities of every species in the dataset.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

A deep learning approach to photo–identification demonstrates high performance on two dozen cetacean species

et al. 2023

Self Cite

View full text Add to dashboard Cite

Researchers can investigate many aspects of animal ecology through noninvasive photo–identification. Photo–identification is becoming more efficient as matching individuals between photos is increasingly automated. However, the convolutional neural network models that have facilitated this change need many training images to generalize well. As a result, they have often been developed for individual species that meet this threshold. These single‐species methods might underperform, as they ignore potential similarities in identifying characteristics and the photo–identification process among species. In this paper, we introduce a multi‐species photo–identification model based on a state‐of‐the‐art method in human facial recognition, the ArcFace classification head. Our model uses two such heads to jointly classify species and identities, allowing species to share information and parameters within the network. As a demonstration, we trained this model with 50,796 images from 39 catalogues of 24 cetacean species, evaluating its predictive performance on 21,192 test images from the same catalogues. We further evaluated its predictive performance with two external catalogues entirely composed of identities that the model did not see during training. The model achieved a mean average precision (MAP) of 0.869 on the test set. Of these, 10 catalogues representing seven species achieved a MAP score over 0.95. For some species, there was notable variation in performance among catalogues, largely explained by variation in photo quality. Finally, the model appeared to generalize well, with the two external catalogues scoring similarly to their species' counterparts in the larger test set. From our cetacean application, we provide a list of recommendations for potential users of this model, focusing on those with cetacean photo–identification catalogues. For example, users with high quality images of animals identified by dorsal nicks and notches should expect near optimal performance. Users can expect decreasing performance for catalogues with higher proportions of indistinct individuals or poor quality photos. Finally, we note that this model is currently freely available as code in a GitHub repository and as a graphical user interface, with additional functionality for collaborative data management, via Happywhale.com.

show abstract

Section: Discussionmentioning

confidence: 99%

“…While this model has been compared to the other 39,284 models in the competition, it has not been explicitly compared to cetacean photo-id models in the literature (e.g. Bergler et al, 2021;Maglietta et al, 2020Maglietta et al, , 2023Thompson et al, 2021;Weideman et al, 2017) (Khan et al, 2020).…”

Section: This Algorithm Is Freely Availablementioning

confidence: 99%

A deep learning approach to photo–identification demonstrates high performance on two dozen cetacean species

et al. 2023

Self Cite

View full text Add to dashboard Cite

show abstract

“…The raw deep learning module also functions as a basis for further development, in order to seamlessly integrate newly trained ANIMAL-SPOT versions based on advanced network architectures such as ResNeXt 62 and HrNet 63 . Another approach that is currently under development is the integration of other viable detection or localization methods such as Memristor based sound localization 64 or FIN-PRINT 65 , a fully automated framework for the individual recognition of killer whales in pictures. Such an addition could introduce new multi-modal possibilities for improvements in detection, localisation and classification.…”

Section: Discussionmentioning

confidence: 99%

ORCA-SPY enables killer whale sound source simulation, detection, classification and localization using an integrated deep learning-based segmentation

Hauer,

Nöth,

Barnhill

et al. 2023

Sci Rep

Self Cite

View full text Add to dashboard Cite

Acoustic identification of vocalizing individuals opens up new and deeper insights into animal communications, such as individual-/group-specific dialects, turn-taking events, and dialogs. However, establishing an association between an individual animal and its emitted signal is usually non-trivial, especially for animals underwater. Consequently, a collection of marine species-, array-, and position-specific ground truth localization data is extremely challenging, which strongly limits possibilities to evaluate localization methods beforehand or at all. This study presents ORCA-SPY, a fully-automated sound source simulation, classification and localization framework for passive killer whale (Orcinus orca) acoustic monitoring that is embedded into PAMGuard, a widely used bioacoustic software toolkit. ORCA-SPY enables array- and position-specific multichannel audio stream generation to simulate real-world ground truth killer whale localization data and provides a hybrid sound source identification approach integrating ANIMAL-SPOT, a state-of-the-art deep learning-based orca detection network, followed by downstream Time-Difference-Of-Arrival localization. ORCA-SPY was evaluated on simulated multichannel underwater audio streams including various killer whale vocalization events within a large-scale experimental setup benefiting from previous real-world fieldwork experience. Across all 58,320 embedded vocalizing killer whale events, subject to various hydrophone array geometries, call types, distances, and noise conditions responsible for a signal-to-noise ratio varying from $$-14.2$$ - 14.2 dB to 3 dB, a detection rate of 94.0 % was achieved with an average localization error of 7.01$$^\circ$$ ∘ . ORCA-SPY was field-tested on Lake Stechlin in Brandenburg Germany under laboratory conditions with a focus on localization. During the field test, 3889 localization events were observed with an average error of 29.19$$^\circ$$ ∘ and a median error of 17.54$$^\circ$$ ∘ . ORCA-SPY was deployed successfully during the DeepAL fieldwork 2022 expedition (DLFW22) in Northern British Columbia, with a mean average error of 20.01$$^\circ$$ ∘ and a median error of 11.01$$^\circ$$ ∘ across 503 localization events. ORCA-SPY is an open-source and publicly available software framework, which can be adapted to various recording conditions as well as animal species.

show abstract

“…Recently, deep-learning-based methods have become a popular means for handling re-identification problems within marine computer vision. Bergler et al proposed a multi-stage deep-learning-based framework for detecting and identifying individual killer whales [41]. They trained and used a YOLOv3 model [42] for detecting dorsal fins and a multi-class classification ResNet-34 model [43] for determining the identity of the killer whales.…”

Section: Related Workmentioning

confidence: 99%

Finding Nemo’s Giant Cousin: Keypoint Matching for Robust Re-Identification of Giant Sunfish

Pedersen

Nyegaard²,

Moeslund

2023

JMSE

View full text Add to dashboard Cite

The Giant Sunfish (Mola alexandrini) has unique patterns on its body, which allow for individual identification. By continuously gathering and matching images, it is possible to monitor and track individuals across location and time. However, matching images manually is a tedious and time-consuming task. To automate the process, we propose a pipeline based on finding and matching keypoints between image pairs. We evaluate our pipeline with four different keypoint descriptors, namely ORB, SIFT, RootSIFT, and SuperPoint, and demonstrate that the number of matching keypoints between a pair of images is a strong indicator for the likelihood that they contain the same individual. The best results are obtained with RootSIFT, which achieves an mAP of 75.91% on our test dataset (TinyMola+) without training or fine-tuning any parts of the pipeline. Furthermore, we show that the pipeline generalizes to other domains, such as re-identification of seals and cows. Lastly, we discuss the impracticality of a ranking-based output for real-life tasks and propose an alternative approach by viewing re-identification as a binary classification. We show that the pipeline can be easily modified with minimal fine-tuning to provide a binary output with a precision of 98% and recall of 44% on the TinyMola+ dataset, which basically eliminates the need for time-consuming manual verification on nearly half the dataset.

show abstract

FIN-PRINT a fully-automated multi-stage deep-learning-based framework for the individual recognition of killer whales

Cited by 11 publications

References 62 publications

A deep learning approach to photo–identification demonstrates high performance on two dozen cetacean species

A deep learning approach to photo–identification demonstrates high performance on two dozen cetacean species

ORCA-SPY enables killer whale sound source simulation, detection, classification and localization using an integrated deep learning-based segmentation

Finding Nemo’s Giant Cousin: Keypoint Matching for Robust Re-Identification of Giant Sunfish

Contact Info

Product

Resources

About