Sound stream segregation: a neuromorphic approach to solve the “cocktail party problem” in real-time

Abstract

The human auditory system has the ability to segregate complex auditory scenes into a foreground component and a background, allowing us to listen to specific speech sounds from a mixture of sounds. Selective attention plays a crucial role in this process, colloquially known as the “cocktail party effect.” It has not been possible to build a machine that can emulate this human ability in real-time. Here, we have developed a framework for the implementation of a neuromorphic sound segregation algorithm in a Field Programmable Gate Array (FPGA). This algorithm is based on the principles of temporal coherence and uses an attention signal to separate a target sound stream from background noise. Temporal coherence implies that auditory features belonging to the same sound source are coherently modulated and evoke highly correlated neural response patterns. The basis for this form of sound segregation is that responses from pairs of channels that are strongly positively correlated belong to the same stream, while channels that are uncorrelated or anti-correlated belong to different streams. In our framework, we have used a neuromorphic cochlea as a frontend sound analyser to extract spatial information of the sound input, which then passes through band pass filters that extract the sound envelope at various modulation rates. Further stages include feature extraction and mask generation, which is finally used to reconstruct the targeted sound. Using sample tonal and speech mixtures, we show that our FPGA architecture is able to segregate sound sources in real-time. The accuracy of segregation is indicated by the high signal-to-noise ratio (SNR) of the segregated stream (90, 77, and 55 dB for simple tone, complex tone, and speech, respectively) as compared to the SNR of the mixture waveform (0 dB). This system may be easily extended for the segregation of complex speech signals, and may thus find various applications in electronic devices such as for sound segregation and speech recognition.

Computational Models of Auditory Scene Analysis: A Review

“…Computational models of auditory scene analysis vary in their fundamental goals; while some attempt to address the complexity that the auditory system faces when processing realistic sounds (such as speech; Nix and Hohmann, 2007; Elhilali and Shamma, 2008; Krishnan et al, 2014; Thakur et al, 2015) in natural environments, others (Wang and Chang, 2008; Boes et al, 2011; Mill et al, 2013; Barniv and Nelken, 2015; Rankin et al, 2015) are built in order to test the potential of some algorithm to simulate specific behavioral and/or neurophysiological experiments. For example, Wang and Chang (2008) measure the fitness of their model based on its ability to reproduce the fission and temporal coherence boundaries reported by van Noorden (1975).…”

“…This representation is used in all of the models in this section, although the dimensions used can vary among them. The neuromorphic model of Thakur et al (2015) is a computationally simplified version of Krishnan et al's (2014) model aiming only to segregate foreground and background, but in real-time. It introduces a formulation of selective attention, which works as an a-priori defined mask on the stimulus representation, selecting a subset of the coincidence matrices for computation.…”

“…Two types of mask formations are implemented. The first type is based on attentional signals, similarly to those in the model of Thakur et al (2015). The other type uses supervised prior training on mask classification, i.e., to classify the target—and non-target masks.…”

“…The model was reported to simulate behavioral results, i.e., it segregated the ABAB sequence and also the complex harmonics as expected from experimental studies (van Noorden, 1975). Speech sounds were also segregated, with the help of an attentional signal, similar to the one that was used in the model of Thakur et al (2015). Comparisons between the two methods for mask formation showed no significant differences between them in this study.…”

The Role of Spatial Separation on Selective and Distributed Attention to Speech

“…Stream segregation depends critically on the distinctiveness of spectro-temporal features of different speakers (Bronkhorst, 2015; Elhilali et al, 2009; Moore and Gockel, 2002; Shamma et al, 2011; Thakur et al, 2015). For example, it is more difficult to segregate two recordings spoken by the same voice vs. those spoken by different speakers (Brungart et al 2001; Ericson et al 2004), and segregation of temporally asynchronous utterances is easier than that of synchronous speech (Humes et al, 2017).…”