Inharmonic speech reveals the role of harmonicity in the cocktail party problem

Abstract

The “cocktail party problem” requires us to discern individual sound sources from mixtures of sources. The brain must use knowledge of natural sound regularities for this purpose. One much-discussed regularity is the tendency for frequencies to be harmonically related (integer multiples of a fundamental frequency). To test the role of harmonicity in real-world sound segregation, we developed speech analysis/synthesis tools to perturb the carrier frequencies of speech, disrupting harmonic frequency relations while maintaining the spectrotemporal envelope that determines phonemic content. We find that violations of harmonicity cause individual frequencies of speech to segregate from each other, impair the intelligibility of concurrent utterances despite leaving intelligibility of single utterances intact, and cause listeners to lose track of target talkers. However, additional segregation deficits result from replacing harmonic frequencies with noise (simulating whispering), suggesting additional grouping cues enabled by voiced speech excitation. Our results demonstrate acoustic grouping cues in real-world sound segregation.

Ecological origins of perceptual grouping principles in the auditory system

“…Our results complement a long research tradition that has documented behavioral and neural effects of a handful of acoustic grouping cues, relying on intuitively plausible cues and synthetic stimuli [7,16,17,18,19,20,21,22,23,38,39,40]. Our results provide statistical justification for the two most commonly studied cues from this literature (onset and harmonicity), but also identify other statistical effects, and show their perceptual relevance.…”

“…This template also detects misalignments in fundamental frequency ( Fig. 6C, middle right), another established grouping cue [16,17,18,19,33].…”

“…We first considered the two most commonly cited cues from traditional accounts of auditory grouping: common onset and offset [20,21,22,23,24,25], and common fundamental frequency [16,17,18,19]. We measured onsets and offsets of each feature as the time points where their broadband envelope exceeded or dropped below a threshold value, and measured the difference in onset or offset time for all pairs of strongly positively or negatively co-activated features (corresponding to the top 75% of positive entries and bottom 75% of negative entries in the association strength tensor).…”

“…1). As in other sensory systems, auditory grouping is believed to exploit acoustic regularities of natural stimuli, such as the tendency of frequencies to be harmonically related [16,17,18,19] or to share a common onset [20,21,22,23,24,25]. But because acoustic grouping cues have traditionally been identified using informal observation and investigated using simple synthetic stimuli, much remains unknown.…”

Development of consonance preferences in Western listeners.

“…The auditory system uses this harmonic information to identify sound sources in complex sound environments (Bregman, 1990). For instance, human listeners detect when a single harmonic is mistuned in a speech signal (Popham et al, 2018) or synthetic tone (Moore et al 1986;Folland, Butler, Smith, & Trainor, 2012;Alain, Arnott, & Picton, 2001), hearing it as coming from a separate source.…”