A speech envelope landmark for syllable encoding in human superior temporal gyrus

Oganian, Yulia; Chang, Edward F.

doi:10.1126/sciadv.aay6279

Cited by 153 publications

(179 citation statements)

References 58 publications

Supporting

Mentioning

157

Contrasting

Order By: Relevance

“…However, we also wanted to explore simpler models to determine the level of complexity that would be required to optimize prediction. Sound onsets offer a promising candidate for a neurally relevant, low-dimensional auditory feature [ 36 , 39 , 40 , 43 ]. As a first test of this hypothesis, we reduced the articulatory features to phoneme onsets (Sg & PhOn, pink).…”

Section: Resultsmentioning

confidence: 99%

“…Another possibility is that the performance boost provided by articulatory features is instead attributable to their correlation with simpler acoustic properties. It has repeatedly been observed that neuronal responses from bilateral superior temporal regions are particularly sensitive to acoustic edges [ 35 , 36 , 37 , 38 , 39 , 40 ]. Features that extract these onsets from envelope representations via a half-wave rectification of the temporal gradient of time-varying energy have been used in several studies [ 36 , 41 , 42 ].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Simple Acoustic Features Can Explain Phoneme-Based Predictions of Cortical Responses to Speech

2019

View full text Add to dashboard Cite

Summary When we listen to speech, we have to make sense of a waveform of sound pressure. Hierarchical models of speech perception assume that, to extract semantic meaning, the signal is transformed into unknown, intermediate neuronal representations. Traditionally, studies of such intermediate representations are guided by linguistically defined concepts, such as phonemes. Here, we argue that in order to arrive at an unbiased understanding of the neuronal responses to speech, we should focus instead on representations obtained directly from the stimulus. We illustrate our view with a data-driven, information theoretic analysis of a dataset of 24 young, healthy humans who listened to a 1 h narrative while their magnetoencephalogram (MEG) was recorded. We find that two recent results, the improved performance of an encoding model in which annotated linguistic and acoustic features were combined and the decoding of phoneme subgroups from phoneme-locked responses, can be explained by an encoding model that is based entirely on acoustic features. These acoustic features capitalize on acoustic edges and outperform Gabor-filtered spectrograms, which can explicitly describe the spectrotemporal characteristics of individual phonemes. By replicating our results in publicly available electroencephalography (EEG) data, we conclude that models of brain responses based on linguistic features can serve as excellent benchmarks. However, we believe that in order to further our understanding of human cortical responses to speech, we should also explore low-level and parsimonious explanations for apparent high-level phenomena.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Simple Acoustic Features Can Explain Phoneme-Based Predictions of Cortical Responses to Speech

2019

View full text Add to dashboard Cite

show abstract

“…One sensible candidate for such a transformation is the first temporal derivative, which is commonly used to augment spectral acoustic features in automatic speech recognition algorithms (Hunt, 1999). Recent data show that neuronal populations in the human auditory do not encode moment-by-moment changes in the amplitude envelope of speech, but rather detect local maxima (i.e., onset edges) in the first derivative of the envelope (Oganian & Chang, 2018). These local maxima often occur during consonant-vowel transitions and, indeed, some of our own work shows that visual speech contributes maximally to audiovisual fusion (McGurk & MacDonald, 1976) during peaks in the first derivative of the interlip distance function (i.e., lip velocity) occurring at a consonant-vowel transition (Jonathan H Venezia, Thurman, Matchin, George, & Hickok, 2015).…”

Section: Discussionmentioning

confidence: 99%

The Role of Multisensory Temporal Covariation in Audiovisual Speech Recognition in Noise

Jh¹,

Sandlin²,

Wojno³

et al. 2019

Preprint

View full text Add to dashboard Cite

Static and dynamic visual speech cues contribute to audiovisual (AV) speech recognition in noise. Static cues (e.g., “lipreading”) provide complementary information that enables perceivers to ascertain ambiguous acoustic-phonetic content. The role of dynamic cues is less clear, but one suggestion is that temporal covariation between facial motion trajectories and the speech envelope enables perceivers to recover a more robust representation of the time-varying acoustic signal. Modeling studies show this is computationally feasible, though it has not been confirmed experimentally. We conducted two experiments to determine whether AV speech recognition depends on the magnitude of cross-sensory temporal coherence (AVC). In Experiment 1, sentence-keyword recognition in steady-state noise (SSN) was assessed across a range of signal-to-noise ratios (SNRs) for auditory and AV speech. The auditory signal was unprocessed or filtered to remove 3-7 Hz temporal modulations. Filtering severely reduced AVC (magnitude-squared coherence of lip trajectories with cochlear-narrowband speech envelopes), but did not reduce the magnitude of the AV advantage (AV > A; ~ 4 dB). This did not depend on the presence of static cues, manipulated via facial blurring. Experiment 2 assessed AV speech recognition in SSN at a fixed SNR (-10.5 dB) for subsets of Exp. 1 stimuli with naturally high or low AVC. A small effect (~ 5% correct; high-AVC > low-AVC) was observed. A computational model of AV speech intelligibility based on AVC yielded good overall predictions of performance, but over-predicted the differential effects of AVC. These results suggest the role and/or computational characterization of AVC must be re-conceptualized.

show abstract

“…At the syllabic level, it has been shown that auditory cortex neuronal oscillations in the theta range are elicited without higher-level speech processing (Howard and Poeppel, 2010;Rimmele et al, 2015), and oscillations occur in sync with acoustic cues at the syllabic scale (Doelling et al, 2014;Oganian and Chang, 2019). At the phrasal level, knowledge about the nature of the acoustic cues that constitute accentuation is sparse, even though linguistic cues in the delta range have been pointed out to be relevant for comprehension (Aubanel et al, 2016).…”

Section: Acoustic Driven Versus Top-down Driven Delta Periodicitiesmentioning

confidence: 99%

“…These models proved to be capable of explaining a range of counterintuitive psychophysical data (e.g., Ghitza and Greenberg, 2009;Ghitza, 2011Ghitza, , 2012Ghitza, , 2014 that are hard to explain by conventional models of speech perception. Doelling et al (2014) provided MEG evidence for this computation principle, showing that perceptual segmentation on the syllabic level appears to require acoustic-driven theta neuronal oscillations (see also: Park et al, 2015;Oganian and Chang, 2019).…”

Section: Introductionmentioning

confidence: 99%

Acoustically driven cortical delta oscillations underpin perceptual chunking

Rimmele

Poeppel

Ghitza

2020

Preprint

View full text Add to dashboard Cite

AbstractOscillation-based models of speech perception postulate a cortical computational principle by which decoding is performed within a window structure derived by a segmentation process. In the syllable level segmentation is realized by a theta oscillator. We provide evidence for an analogous role of a delta oscillator at the phrasal level. We recorded MEG while participants performed a target identification task. Random-digit strings, with phrasal chunks of two digits, were presented at chunk rates inside or outside of the delta range. Strong periodicities were elicited by acoustic-driven chunk rates inside of delta, in superior and middle temporal areas and speech-motor integration areas. Periodicities were diminished or absent for chunk rates outside of the delta range, closely in line with behavioral performance. No periodicities were observed for top-down driven chunking conditions. Our findings show that phrasal chunking is correlated with acoustic-driven delta oscillations, expressing anatomically specific patterns of neuronal periodicities.

show abstract

A speech envelope landmark for syllable encoding in human superior temporal gyrus

Cited by 153 publications

References 58 publications

Simple Acoustic Features Can Explain Phoneme-Based Predictions of Cortical Responses to Speech

Simple Acoustic Features Can Explain Phoneme-Based Predictions of Cortical Responses to Speech

The Role of Multisensory Temporal Covariation in Audiovisual Speech Recognition in Noise

Acoustically driven cortical delta oscillations underpin perceptual chunking

Contact Info

Product

Resources

About