Interactive extraction of diverse vocal units from a planar embedding without the need for prior sound segmentation

Abstract

Annotating and proofreading data sets of complex natural behaviors such as vocalizations are tedious tasks because instances of a given behavior need to be correctly segmented from background noise and must be classified with minimal false positive error rate. Low-dimensional embeddings have proven very useful for this task because they can provide a visual overview of a data set in which distinct behaviors appear in different clusters. However, low-dimensional embeddings introduce errors because they fail to preserve distances; and embeddings represent only objects of fixed dimensionality, which conflicts with vocalizations that have variable dimensions stemming from their variable durations. To mitigate these issues, we introduce a semi-supervised, analytical method for simultaneous segmentation and clustering of vocalizations. We define a given vocalization type by specifying pairs of high-density regions in the embedding plane of sound spectrograms, one region associated with vocalization onsets and the other with offsets. We demonstrate our two-neighborhood (2N) extraction method on the task of clustering adult zebra finch vocalizations embedded with UMAP. We show that 2N extraction allows the identification of short and long vocal renditions from continuous data streams without initially committing to a particular segmentation of the data. Also, 2N extraction achieves much lower false positive error rate than comparable approaches based on a single defining region. Along with our method, we present a graphical user interface (GUI) for visualizing and annotating data.

Multimodal system for recording individual-level behaviors in songbird groups

“…While there are many methods for single channel segmentation available [38][39][40] , only few combine information from multiple sensor channels 40 and none has made use of accelerometer data so far. Also, it remains open how well vocalizations can be separated from the ve stationary microphones alone.…”

Multimodal system for recording individual-level behaviors in songbird groups

“…To enable automatic segmentation of longitudinal BirdPark data, new methods will need to be developed. While there are many methods for single channel segmentation available [38][39][40] , only few combine information from multiple sensor channels 40 and none has made use of accelerometer data so far.…”

Benchmarking nearest neighbor retrieval of zebra finch vocalizations across development

“…We annotated vocal segments (not further classified into vocalization types) with high temporal accuracy. To generate these gold-standard (GS) annotations, we used a semi-supervised segmentation method 13 , correcting poor segments and eliminating false positives by visual inspection of spectrograms. To eliminate false negatives, the present NN method was used with the cosine distance as metric.…”

“…Entirely lacking are public datasets of precisely segmented subsongs; a recent massive-data study on this important developmental phase 12 simply ignores the segmentation problem and takes as proxy of vocalizations all amplitude-thresholded sound segments, semi-automatically excluding false positives in such a way to introduce false negatives (see Appendix). Unfortunately, amplitude thresholding can create severe problems if the recording quality is low 13 , which only emphasizes that this severe lack of training and test data forms a bottleneck for progress in large-scale research on vocal development, and it calls for the creation of gold-standard data sets.…”

“…One method for bootstrapping large vocal data sets from few precisely labelled samples is nearest neighbor (NN) retrieval 13 . NN retrieval is a highly successful information retrieval method 14 : it is used in tasks such as tagging images 15 , web mining 16 , recommendation systems 17,18 , and for inference in language models 19,20 .…”