Latent space visualization, characterization, and generation of diverse vocal communication signals

Abstract

Animals produce vocalizations that range in complexity from a single repeated call to hundreds 1 of unique vocal elements patterned in sequences unfolding over hours. Characterizing complex 2 vocalizations can require considerable effort and a deep intuition about each species' vocal behavior. 3 Even with a great deal of experience, human characterizations of animal communication can be 4 affected by human perceptual biases. We present here a set of computational methods that center 5 around projecting animal vocalizations into low dimensional latent representational spaces that 6 are directly learned from data. We apply these methods to diverse datasets from over 20 species, 7 including humans, bats, songbirds, mice, cetaceans, and nonhuman primates, enabling high-powered 8 comparative analyses of unbiased acoustic features in the communicative repertoires across species. 9 Latent projections uncover complex features of data in visually intuitive and quantifiable ways. We 10 introduce methods for analyzing vocalizations as both discrete sequences and as continuous latent 11 variables. Each method can be used to disentangle complex spectro-temporal structure and observe 12 long-timescale organization in communication. Finally, we show how systematic sampling from 13 latent representational spaces of vocalizations enables comprehensive investigations of perceptual 14 and neural representations of complex and ecologically relevant acoustic feature spaces. 15Of the thousands of species that communicate vocally, the repertoires of only a tiny minority have been characterized or 18 studied in detail. This is due, in large part, to traditional analysis methods that require a high level of expertise that is 19 hard to develop and often species-specific. Here, we present a set of novel methods to project animal vocalizations 20 into latent feature spaces to quantitatively compare and develop visual intuitions about animal vocalizations, and to 21 systematically synthesize novel species-typical vocalizations from learned feature sets. We demonstrate these methods 22 across a series of analyses over 19 datasets of animal vocalizations from 29 different species, including songbirds, mice, 23 monkeys, humans, and whales. We show how learned latent feature spaces untangle complex spectro-temporal structure, 24 enable unbiased comparisons, and uncover high-level features such as individual identity and population dialects. We 25 generate smoothly varying morphs between vocalizations from a songbird species with a spectro-temporally complex 26 vocal repertoire, European starlings, and show how these methods enable a new degree of control over ecologically 27 relevant signals that can be broadly applied across behavioral and physiological experimental settings. 28 2 Introduction 29Vocal communication is a social behavior common to much of the animal kingdom in which acoustic signals are 30 transmitted from sender to receiver to convey various forms of information such as identity, individual fitness, or the 31 presence ...

Uniform Manifold Approximation and Projection for Clustering Taxa through Vocalizations in a Neotropical Passerine (Rough-Legged Tyrannulet, Phyllomyias burmeisteri)

“…In that scenario, as our results suggested, each successive dimension represents an orthogonal transformation capturing the maximum possible variance between the data [68,70]. Vocalizations can also be decomposed into characteristics using linear discriminant analysis [5] where characteristics are determined by their ability to explain variance in a specific dimension of data, such as individual identity [68,71]. However, the dimensionality reduction may also be non-linear, representing a greater number of data relationships (for example, the similarity between notes of birdsong).…”

“…For instance, species that due to their morphological characteristics as plumage are difficult to separate taxonomically. However, it is important to apply and define this technique in organisms with more complex vocalizations, taking into account that animals emit acoustic signals that vary in complexity, which represents from a single repeated call (note), as in the case of our species model, to hundreds of different vocal elements [68].…”

“…Animal vocalizations have been previously projected onto a linear feature of spaces using principal component analysis [10,69]. In that scenario, as our results suggested, each successive dimension represents an orthogonal transformation capturing the maximum possible variance between the data [68,70]. Vocalizations can also be decomposed into characteristics using linear discriminant analysis [5] where characteristics are determined by their ability to explain variance in a specific dimension of data, such as individual identity [68,71].…”

TweetyNet: A neural network that enables high-throughput, automated annotation of birdsong

“…Aspects of other deep networks applied to animal motor control may improve TweetyNet. Examples include object detection architectures [47,48] applied to mouse ultrasonic vocalizations and animal motion tracking, and generative architectures applied to birdsong and other vocalizations [49][50][51]. Lastly we note that in principle TweetyNet and vak library can be applied to any other annotated vocalization, including calls of bats, mouse ultrasonic vocalizations, and dolphin communication.…”

Fine‐scale observations of spatio‐spectro‐temporal dynamics of bird vocalizations using robot audition techniques

“…The application of dimension reduction algorithms or latent space visualization of communication signals is also getting much attention in the wide field of bioacoustics. Sainburg proposed a package for this purpose, called animal vocalization generative network (AVGN) (Sainburg et al 2019). They showed the various examples of the use of the latent space such as discrete latent projections of animal vocalizations and temporally continuous latent trajectories.…”