TweetyNet: A neural network that enables high-throughput, automated annotation of birdsong

Abstract

Songbirds provide an excellent model system for understanding sensorimotor learning. Many analyses of learning require annotating song, but songbirds produce more songs than can be annotated by hand. Existing methods for automating annotation are challenged by variable song, like that of Bengalese finches. For particularly complex song like that of canaries, no methods exist, limiting the questions researchers can investigate. We developed an artificial neural network, TweetyNet, that automates annotation. First we benchmark the network on open datasets of Bengalese finch song, showing that TweetyNet achieves significantly lower error than a similar method, using less training data, and maintains low error across multiple days of song. We then show TweetyNet performs similarly on canary song. This accuracy allowed fully-automated analyses of datasets an order of magnitude larger than previous studies, improved the precision of statistical models of syntax, and revealed novel details of syntax in a new canary strain. Hence TweetyNet enables automated annotation and analysis of Bengalese finch and canary song that was formerly manual.

Canary Song Decoder: Transduction and Implicit Segmentation with ESNs and LTSMs

“…MFCC computations where performed using the Librosa [13] Python library. Song spectrograms are first extracted using Short Time Fourier Transform every 11ms (often called frame stride) and computed on overlapping windows of 23ms (often called window width) 4 , using a Hanning window to reduce edge effects. Then, we set the frequency range of a 128 filters Mel filterbank to [500Hz; 8kHz], as canaries vocal patterns occur below 8kHz and as the [0Hz; 500Hz] bandwidth represents mostly noise.…”

“…Comparison with [4] can not be done fairly, as our method operate at phrase level and not at syllable level, and as we did not use the same dataset. We discuss the possibility of extending this work in Discussion.…”

“…Ongoing work described in Cohen et. al [4] solves the segmentation problem by using machine learning techniques to try to extract the position of syllables on spectrograms of the song while classifying them using CNN and Long Short Term Memory Recurrent Neural Networks (LSTM RNN).…”

“…We propose a method based on RNN processing spectral features to segment and classify canary songs. However, unlike [4], we focused on segmenting canary songs at phrase level, which is supposed to be sufficient for analysis like [12]. We compare simple neural architectures like a single LSTM or Echo States Networks (ESN) [7].…”

“…syllables types). Importantly, the limited number of parameters that need to be learned (compared to more complex architectures like [4]) enables one to apply them on limited amount of data while not overfitting on it; this is particularly interesting because it limits the amount of data necessary to be hand-labelled.…”