SongExplorer: A deep learning workflow for discovery and segmentation of animal acoustic communication signals

Abstract

Many animals produce distinct sounds or substrate-borne vibrations, but these signals have proved challenging to segment with automated algorithms. We have developed SongExplorer, a web-browser based interface wrapped around a deep-learning algorithm that supports an interactive workflow for (1) discovery of animal sounds, (2) manual annotation, (3) supervised training of a deep convolutional neural network, and (4) automated segmentation of recordings. Raw data can be explored by simultaneously examining song events, both individually and in the context of the entire recording, watching synced video, and listening to song. We provide a simple way to visualize many song events from large datasets within an interactive low-dimensional visualization, which facilitates detection and correction of incorrectly labelled song events. The machine learning model we implemented displays higher accuracy than existing heuristic algorithms and similar accuracy as two expert human annotators. We show that SongExplorer allows rapid detection of all song types from new species and of novel song types in previously well-studied species.

Fast and accurate annotation of acoustic signals with deep neural networks

“…DAS is a supervised annotation method: It discriminates syllable types that have been manually assigned different labels during training. By contrast, unsupervised methods can determine in unlabelled data whether syllables fall into distinct types and if so, classify the syllables ( Tabler et al, 2017 ; Coffey et al, 2019 ; Clemens et al, 2018 ; Goffinet et al, 2021 ; Sainburg et al, 2020 ; Sangiamo et al, 2020 ; Arthur et al, 2021 ). While DAS does not require large amounts of manual annotations ( Figure 4D ), manual labeling of syllable types can be tedious when differences between syllable types are subtle ( Clemens et al, 2018 ) or when repertoires are large ( Sangiamo et al, 2020 ).…”

“…We show that the annotation burden can be further reduced using unsupervised classification of syllable types, in particular for species with large or individual-specific repertoires ( Figure 5 , Clemens et al, 2018 ; Tabler et al, 2017 ; Coffey et al, 2019 ; Goffinet et al, 2021 ; Sainburg et al, 2020 ; Arthur et al, 2021 ). In the future, incorporating recent advances in the self-supervised or semi-supervised training of neural networks will likely further reduce data requirements ( Mathis et al, 2021 ; Raghu et al, 2019 ; Devlin et al, 2019 ; Chen and He, 2020 ).…”

Fast and accurate annotation of acoustic signals with deep neural networks

“…DAS is a supervised annotation method: It discriminates syllable types that have been manually assigned different labels during training. By contrast, unsupervised methods can determine in unlabelled data whether syllables fall into distinct types and if so, classify the syllables ( Tabler et al, 2017 ; Coffey et al, 2019 ; Clemens et al, 2018 ; Goffinet et al, 2021 ; Sainburg et al, 2020 ; Sangiamo et al, 2020 ; Arthur et al, 2021 ). While DAS does not require large amounts of manual annotations ( Figure 4D ), manual labeling of syllable types can be tedious when differences between syllable types are subtle ( Clemens et al, 2018 ) or when repertoires are large ( Sangiamo et al, 2020 ).…”

“…We show that the annotation burden can be further reduced using unsupervised classification of syllable types, in particular for species with large or individual-specific repertoires ( Figure 5 ) ( Clemens et al, 2018 ; Tabler et al, 2017 ; Coffey et al, 2019 ; Goffinet et al, 2021 ; Sainburg et al, 2020 ; Arthur et al, 2021 ). In the future, incorporating recent advances in the self-supervised or semi-supervised training of neural networks will likely further reduce data requirements ( Mathis et al, 2021 ; Raghu et al, 2019 ; Devlin et al, 2019 ; Chen and He, 2020 ).…”

“…These methods are not only fast and accurate but also easily adapted to novel signals by non-experts since they only require annotated examples for learning. Recently, deep neural networks have also been used for annotating animal vocalizations ( Oikarinen et al, 2019 ; Coffey et al, 2019 ; Cohen et al, 2020 ; Sainburg et al, 2020 ; Arthur et al, 2021 ; Goffinet et al, 2021 ).…”