Automatic Recognition of Element Classes and Boundaries in the Birdsong with Variable Sequences

Abstract

Researches on sequential vocalization often require analysis of vocalizations in long continuous sounds. In such studies as developmental ones or studies across generations in which days or months of vocalizations must be analyzed, methods for automatic recognition would be strongly desired. Although methods for automatic speech recognition for application purposes have been intensively studied, blindly applying them for biological purposes may not be an optimal solution. This is because, unlike human speech recognition, analysis of sequential vocalizations often requires accurate extraction of timing information. In the present study we propose automated systems suitable for recognizing birdsong, one of the most intensively investigated sequential vocalizations, focusing on the three properties of the birdsong. First, a song is a sequence of vocal elements, called notes, which can be grouped into categories. Second, temporal structure of birdsong is precisely controlled, meaning that temporal information is important in song analysis. Finally, notes are produced according to certain probabilistic rules, which may facilitate the accurate song recognition. We divided the procedure of song recognition into three sub-steps: local classification, boundary detection, and global sequencing, each of which corresponds to each of the three properties of birdsong. We compared the performances of several different ways to arrange these three steps. As results, we demonstrated a hybrid model of a deep convolutional neural network and a hidden Markov model was effective. We propose suitable arrangements of methods according to whether accurate boundary detection is needed. Also we designed the new measure to jointly evaluate the accuracy of note classification and boundary detection. Our methods should be applicable, with small modification and tuning, to the songs in other species that hold the three properties of the sequential vocalization.

TweetyNet: A neural network that enables high-throughput, automated annotation of birdsong

“…if one bird sang more syllables per bout than another). Measuring the syllable error rate confirmed that TweetyNet consistently achieved similar error rates across the ten birds, as shown in Fig 4B. Because this metric was also used in [38] (as "note error rate"), we can compare our results directly to theirs. As indicated by blue circles in Fig 4B, the best-performing models in that study achieved syllable error rates of 0.83 and 0.46 with two and eight minutes of training data, respectively.…”

“…We first set out to test whether our network robustly annotates syllables across a large number of individual birds. To do so, we made use of the publicly available repository of Bengalese Finch song [39], used to benchmark hybrid neural network-HMM models from [38] as referenced in Proposed Method and Related Work. The repository contains song from 10 individual birds, with hundreds of bouts of hand-annotated song for each bird.…”

“…In summary, the key prior art is the important work of Koumura and Okanoya [38]. This work anticipates the overall structure of our model, but through the integration of multiple distinct components that are individually optimized.…”

“…Below we show that TweetyNet meets our criteria for an algorithm that learns features from data and does not require segmented song to make predictions. To do so we we benchmark TweetyNet on Bengalese finch and canary song, and where possible compare the performance to [38]. Additionally we show that we achieve robust performance: across songs of individuals, which can vary widely even within a species; across many bouts of song from one individual, e.g.…”

“…The architecture that we develop is most directly related to those that have been used for event detection in audio and video [34,35] and for phoneme classification and sequence labeling [33,37]. The closest prior model for segmenting and labeling birdsong is [38]. Several aspects of that study provide context for the contributions of our work.…”