Learning Midlevel Auditory Codes from Natural Sound Statistics

Abstract

Interaction with the world requires an organism to transform sensory signals into representations in which behaviorally meaningful properties of the environment are made explicit. These representations are derived through cascades of neuronal processing stages in which neurons at each stage recode the output of preceding stages. Explanations of sensory coding may thus involve understanding how low-level patterns are combined into more complex structures. To gain insight into such midlevel representations for sound, we designed a hierarchical generative model of natural sounds that learns combinations of spectrotemporal features from natural stimulus statistics. In the first layer, the model forms a sparse convolutional code of spectrograms using a dictionary of learned spectrotemporal kernels. To generalize from specific kernel activation patterns, the second layer encodes patterns of time-varying magnitude of multiple first-layer coefficients. When trained on corpora of speech and environmental sounds, some second-layer units learned to group similar spectrotemporal features. Others instantiate opponency between distinct sets of features. Such groupings might be instantiated by neurons in the auditory cortex, providing a hypothesis for midlevel neuronal computation.

Ecological origins of perceptual grouping principles in the auditory system

“…In order to study grouping in natural sound signals without relying on a prior hypothesis of the features or principles that would be involved, we used convolutional sparse coding [26,27] to first learn a set of features from which natural sounds can be composed. These features were learned from recordings of single sources of speech or musical instruments represented as 'cochleagrams' -time-frequency decompositions intended to approximate the representation of sound in the human cochlea.…”

“…Unlike previous work, our approach was independent of prior hypotheses about the underlying features or the regularities that might relate to grouping. We first derived a set of primitive auditory patterns by learning a dictionary of spectrotemporal features from a corpus of natural sounds, using sparse convolutional coding [26,27]. We then measured co-occurrence statistics for these features in natural sounds.…”

“…In principle the inference of sources from feature activations could be constrained by the full joint distribution of all features. In practice this distribution is challenging to learn and to analyze [27]. Instead, we measured dependencies between pairs of features, which are tractable to measure and analyze, and which we found to contain rich structure.…”

“…A full account of auditory scene analysis will undoubtedly require more complete statistical models of natural sound sources, incorporating more than the pairwise dependencies between local features studied here [27]. In addition to multi-element dependencies, a full model will likely require additional hierarchical structure, in which groups of local features are in turn grouped into larger-scale configurations.…”

“…To learn an acoustic feature basis for cochleagrams we used a convolutional sparse-coding model described in [27] with an additional non-negativity constraint imposed on the basis functions, to aid interpretability in terms of sound energy (which is non-negative) and produce localized features [28]. The model represents a cochleagram excerpt as a sum of spectrotemporal kernels (STKs) ϕ (162 ms in duration) convolved with their activation time courses s:…”