Low-dimensional learned feature spaces quantify individual and group differences in vocal repertoires

Abstract

Vocalization is an essential medium for social and sexual signaling in most birds and mammals. 1 Consequently, the analysis of vocal behavior is of great interest to fields such as neuroscience and 2 linguistics. A standard approach to analyzing vocalization involves segmenting the sound stream 3 into discrete vocal elements, calculating a number of handpicked acoustic features, and then using 4 the feature values for subsequent quantitative analysis. While this approach has proven powerful, 5 it suffers from several crucial limitations: First, handpicked acoustic features may miss important 6 dimensions of variability that are important for communicative function. Second, many analyses 7 assume vocalizations fall into discrete vocal categories, often without rigorous justification. Third, a 8 syllable-level analysis requires a consistent definition of syllable boundaries, which is often difficult 9 to maintain in practice and limits the sorts of structure one can find in the data. To address these 10 shortcomings, we apply a data-driven approach based on the variational autoencoder (VAE), an 11 unsupervised learning method, to the task of characterizing vocalizations in two model species: 12 the laboratory mouse (Mus musculus) and the zebra finch (Taeniopygia guttata). We find that the 13 VAE converges on a parsimonious representation of vocal behavior that outperforms handpicked 14 acoustic features on a variety of common analysis tasks, including representing acoustic similarity and 15 recovering a known effect of social context on birdsong. Additionally, we use our learned acoustic 16 features to argue against the widespread view that mouse ultrasonic vocalizations form discrete 17 syllable categories. Lastly, we present a novel "shotgun VAE" that can quantify moment-by-moment 18 variability in vocalizations. In all, we show that data-derived acoustic features confirm and extend 19 existing approaches while offering distinct advantages in several critical applications. 20 1 Introduction 21 Vocalization is an essential medium for social and sexual signaling in most birds and mammals, and also serves as a 22 natural substrate for language and music in humans. Consequently, the analysis of vocal behavior is of great interest to 23 ethologists, psychologists, linguists, and neuroscientists. A major goal of these various lines of enquiry is to develop 24 methods for quantitative analysis of vocal behavior, efforts that have resulted in several powerful methods that enable 25 the automatic or semi-automatic analysis of vocalizations. Key to this approach has been the existence of software 26 packages that calculate acoustic features for each syllable within a vocalization [4, 39, 40, 7, 6]. For example, Sound 27 Analysis Pro, focused on birdsong, calculates 14 features for each syllable, including duration, spectral entropy, and 28 goodness of pitch, and uses these as a basis for subsequent clustering and analysis [39]. More recently, MUPET and 29 DeepSqueak have applied a similar approach to mouse vocalizati...

Circuit and synaptic organization of forebrain-to-midbrain pathways that promote and suppress vocalization

“…Given the artificial nature of optogenetic stimulation, we wondered whether USVs elicited by 167 optogenetic activation of POA neurons were acoustically similar to the USVs that are normally 168 produced by mice during social interactions. To compare the acoustic features of 169 optogenetically-elicited USVs (opto-USVs) to those of USVs produced spontaneously to a 170 nearby female, we employed a recently described method using variational autoencoders (VAEs) (Goffinet, 2019;Sainburg et al, 2019). Briefly, the VAE is an unsupervised modeling 172 approach that uses spectrograms of vocalizations as inputs and from these data learns a pair of 173 probabilistic maps, an "encoder" and a "decoder," capable of compressing vocalizations into a 174 small number of latent features while attempting to preserve as much information as possible 175 ( Fig.…”

“…3C, right). To quantify the difference between female-directed and opto-USVs for 184 each mouse, we estimated the Maximum Mean Discrepancy (Gretton, 2012) between 185 distributions of latent syllable representations as in (Goffinet, 2019). In addition, a baseline level 186 of variability in syllable repertoire was established for each mouse by estimating the MMD 187 between the first and second halves of female-directed USVs emitted in a recording session 188 unusual opto-USVs tended to be very loud and also had high frequency bandwidth.…”

“…To visualize these 5-dimensional spaces, the 581 latent representations of syllable spectrograms are projected into two dimensions using the 582 UMAP algorithm (McInnes, 2018). To quantify differences in syllable repertoires, we estimate 583 the Maximum Mean Discrepancy (Gretton, 2012) between distributions of latent syllable 584 representations as in (Goffinet, 2019). First, a baseline level of variability in syllable repertoire 585 was established for each mouse by estimating MMD between the first and second halves of 586 female-directed syllables emitted in a recording session.…”

Finding, visualizing, and quantifying latent structure across diverse animal vocal repertoires

“…The utility of non-linear dimensionality reduction techniques are just now coming to fruition in the study of animal communication, for example using t-distributed stochastic neighborhood embedding (t-SNE; [ 32 ]) to describe the development of zebra finch song [ 34 ], using Uniform Manifold Approximation and Projection (UMAP; [ 31 ]) to describe and infer categories in birdsong [ 3 , 35 ], or using deep neural networks to synthesize naturalistic acoustic stimuli [ 36 , 37 ]. Developments in non-linear representation learning have helped fuel the most recent advancements in machine learning, untangling statistical relationships in ways that provide more explanatory power over data than traditional linear techniques [ 13 , 14 ].…”

TweetyNet: A neural network that enables high-throughput, automated annotation of birdsong

“…Aspects of other deep networks applied to animal motor control may improve TweetyNet. Examples include object detection architectures [47,48] applied to mouse ultrasonic vocalizations and animal motion tracking, and generative architectures applied to birdsong and other vocalizations [49][50][51]. Lastly we note that in principle TweetyNet and vak library can be applied to any other annotated vocalization, including calls of bats, mouse ultrasonic vocalizations, and dolphin communication.…”