Fast and accurate annotation of acoustic signals with deep neural networks

Abstract

Acoustic signals serve communication within and across species throughout the animal kingdom. Studying the genetics, evolution, and neurobiology of acoustic communication requires annotating acoustic signals: segmenting and identifying individual acoustic elements like syllables or sound pulses. To be useful, annotations need to be accurate, robust to noise, fast. We introduce DeepSS, a method that annotates acoustic signals across species based on a deep-learning derived hierarchical presentation of sound. We demonstrate the accuracy, robustness, and speed of DeepSS using acoustic signals with diverse characteristics: courtship song from flies, ultrasonic vocalizations of mice, and syllables with complex spectrotemporal structure from birds. DeepSS comes with a graphical user interface for annotating song, training the network, and for generating and proofreading annotations (available at https://janclemenslab.org/deepss). The method can be trained to annotate signals from new species with little manual annotation and can be combined with unsupervised methods to discover novel signal types. DeepSS annotates song with high throughput and low latency, allowing realtime annotations for closed-loop experimental interventions.

Analysis of Mouse Vocal Communication (AMVOC): A deep, unsupervised method for rapid detection, analysis, and classification of ultrasonic vocalizations

“…A more recent tool is Deep Song Segmenter (DeepSS), which has been used for annotation of songs of mice, birds and flies (Steinfath et al, 2021). DeepSS learns a representation of sounds features directly from raw audio recordings using temporal convolutional networks (TCNs), based on dilated convolutions.…”

“…Over the past two decades there has been a growing interest in the usage and signaling of vocalizations in mice, with large efforts going into studying the underlying neurobiological mechanisms for auditory processing (Pomerantz et al, 1983;Liu et al, 2003;Neilans et al, 2014;Perrodin et al, 2020;Holy and Guo, 2005), and the production of vocalizations (Arriaga et al, 2012;Chabout et al, 2016;Okobi et al, 2019;Zimmer et al, 2019;Gao et al, 2019;Tschida et al, 2019;Michael et al, 2020). The tools available for experiments in mice provide a promising model for studying the neural basis of vocalizations, as well as the effects of genes on the origin and development of vocal and neural anatomy (Grimsley et al, 2011;Bowers et al, 2013;Chabout et al, 2016;Tabler et al, The copyright holder for this preprint this version posted August 13, 2021. ; https://doi.org/10.1101/2021.08.13.456283 doi: bioRxiv preprint (Steinfath et al, 2021). DeepSS learns a representation of sounds features directly from raw audio recordings using temporal convolutional networks (TCNs), based on dilated convolutions.…”