Low-level articulatory synthesis: A working text-to-speech solution and a linguistic tool

Hill, David R.; Taube-Schock, Craig; Manzara, Leonard C.

doi:10.1017/cnj.2017.15

Cited by 9 publications

(7 citation statements)

References 51 publications

(67 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Curiously, it becomes no less challenging as we attempt to break spoken language usage into its smallest bits. For instance, even at the fine scale of phonemes pairing individual vowels with individual consonants, speech modeling continues to the present day to struggle with “coarticulation” (Hill et al, 2017 ), i.e., that the physiological production of each phoneme changes depending on its context in the speech stream. Human listeners manage to compensate for coarticulation to the point of responding systematically to audible changes with context—even without explicitly noticing the context effects and even, instead, with having the experience of stable phonemic recognition (Zamuner et al, 2016 ; Viswanathan and Kelty-Stephen, 2018 ).…”

Section: Introductionmentioning

confidence: 99%

Bringing the Nonlinearity of the Movement System to Gestural Theories of Language Use: Multifractal Structure of Spoken English Supports the Compensation for Coarticulation in Human Speech Perception

Ward

Kelty-Stephen

2018

Front. Physiol.

View full text Add to dashboard Cite

Coarticulation is the tendency for speech vocalization and articulation even at the phonemic level to change with context, and compensation for coarticulation (CfC) reflects the striking human ability to perceive phonemic stability despite this variability. A current controversy centers on whether CfC depends on contrast between formants of a speech-signal spectrogram—specifically, contrast between offset formants concluding context stimuli and onset formants opening the target sound—or on speech-sound variability specific to the coordinative movement of speech articulators (e.g., vocal folds, postural muscles, lips, tongues). This manuscript aims to encode that coordinative-movement context in terms of speech-signal multifractal structure and to determine whether speech's multifractal structure might explain the crucial gestural support for any proposed spectral contrast. We asked human participants to categorize individual target stimuli drawn from an 11-step [ga]-to-[da] continuum as either phonemes “GA” or “DA.” Three groups each heard a specific-type context stimulus preceding target stimuli: either real-speech [al] or [a], sine-wave tones at the third-formant offset frequency of either [al] or [aɹ], and either simulated-speech contexts [al] or [aɹ]. Here, simulating speech contexts involved randomizing the sequence of relatively homogeneous pitch periods within vowel-sound [a] of each [al] and [aɹ]. Crucially, simulated-speech contexts had the same offset and extremely similar vowel formants as and, to additional naïve participants, sounded identical to real-speech contexts. However, randomization distorted original speech-context multifractality, and effects of spectral contrast following speech only appeared after regression modeling of trial-by-trial “GA” judgments controlled for context-stimulus multifractality. Furthermore, simulated-speech contexts elicited faster responses (like tone contexts do) and weakened known biases in CfC, suggesting that spectral contrast depends on the nonlinear interactions across multiple scales that articulatory gestures express through the speech signal. Traditional mouse-tracking behaviors measured as participants moved their computer-mouse cursor to register their “GA”-or-“DA” decisions with mouse-clicks suggest that listening to speech leads the movement system to resonate with the multifractality of context stimuli. We interpret these results as shedding light on a new multifractal terrain upon which to found a better understanding in which movement systems play an important role in shaping how speech perception makes use of acoustic information.

show abstract

Section: Introductionmentioning

confidence: 99%

Bringing the Nonlinearity of the Movement System to Gestural Theories of Language Use: Multifractal Structure of Spoken English Supports the Compensation for Coarticulation in Human Speech Perception

Ward

Kelty-Stephen

2018

Front. Physiol.

View full text Add to dashboard Cite

show abstract

“…Both speech production and language perception exemplify just such nonlinear interactions across timescales. Longer-term speech sequences reshape the brief articulation of a single syllable [5]. Communicative sounds exhibit a range of fractional power law exponents [44].…”

Section: Multifractality To Portray Cross-scale Interactions Blending Fleeting Behaviours With Longer-term Behavioursmentioning

confidence: 99%

“…Motor processes reshape phonemes in context- and sequence-dependent ways (e.g. in coarticulation) that text-to-speech synthesis has struggled to emulate [5]. Speech sounds reflect movements beyond articulators to distal parts of the body [6]—potentially explaining their utility for diagnosing Parkinson's disease [7].…”

Section: Introductionmentioning

confidence: 99%

Perceiving and remembering speech depend on multifractal nonlinearity in movements producing and exploring speech

Bloomfield

Lane

Mangalam

et al. 2021

J. R. Soc. Interface.

View full text Add to dashboard Cite

Speech perception and memory for speech require active engagement. Gestural theories have emphasized mainly the effect of speaker's movements on speech perception. They fail to address the effects of listener movement, focusing on communication as a boundary condition constraining movement among interlocutors. The present work attempts to break new ground by using multifractal geometry of physical movement as a common currency for supporting both sides of the speaker–listener dyads. Participants self-paced their listening to a narrative, after which they completed a test of memory querying their narrative comprehension and their ability to recognize words from the story. The multifractal evidence of nonlinear interactions across timescales predicted the fluency of speech perception. Self-pacing movements that enabled listeners to control the presentation of speech sounds constituted a rich exploratory process. The multifractal nonlinearity of this exploration supported several aspects of memory for the perceived spoken language. These findings extend the role of multifractal geometry in the speaker's movements to the narrative case of speech perception. In addition to posing novel basic research questions, these findings make a compelling case for calibrating multifractal structure in text-to-speech synthesizers for better perception and memory of speech.

show abstract

“…Speech perception critically depends on the articulatory gestures shaping those acoustic features [4][5][6]. Human motor processes reshape phonemes in a context-and sequence-dependent way (e.g., in coarticulation) that text-to-speech synthesis has struggled to emulate [7]. Speech sounds reflect movements beyond articulators to distal parts of the body [8]-potentially explaining their utility for diagnosing Parkinson's disease [9].…”

Section: Introductionmentioning

confidence: 99%

“…Multifractality nonlinearity t MF may predict expressiveness [44]. Longer-term speech sequences reshape the brief articulation of a single syllable [7]. Speech perception adjusts to compensate for coarticulatory interaction across time [4] and tailors these compensations over longer timescales [45].…”

Section: Introductionmentioning

confidence: 99%

Perceiving and remembering speech depend on multifractal nonlinearity in movements producing and exploring speech

Bloomfield

Lane²,

Mangalam³

et al. 2021

Preprint

View full text Add to dashboard Cite

Speech perception and memory for speech require active engagement. Gestural theories have emphasized mainly the effect of movements of the speaker on speech perception. They fail to address the effects of listener movement, focusing on communication as a boundary condition constraining movement among interlocutors. The present work attempts to break new ground by using multifractal geometry of physical action as a common currency for supporting both sides of the speaker-listener dyads. Participants self-paced their listening to a narrative, after which they completed a test of memory querying their narrative comprehension and the ability to recognize words from the story. The multifractal evidence of nonlinear interactions across timescales predicted the fluency of speech perception. Self-pacing movements that enabled listeners to control the presentation of speech sounds constituted a rich exploratory process. The multifractal nonlinearity of this exploration supported several aspects of memory for the perceived spoken language. These findings extend the role of multifractal geometry in the actions of the speaker to the narrative case of speech perception. In addition to posing novel basic research questions, these findings make a compelling case for imbuing multifractal structure in text-to-speech synthesizers for better perception and memory of speech.

show abstract

Low-level articulatory synthesis: A working text-to-speech solution and a linguistic tool

Cited by 9 publications

References 51 publications

Bringing the Nonlinearity of the Movement System to Gestural Theories of Language Use: Multifractal Structure of Spoken English Supports the Compensation for Coarticulation in Human Speech Perception

Bringing the Nonlinearity of the Movement System to Gestural Theories of Language Use: Multifractal Structure of Spoken English Supports the Compensation for Coarticulation in Human Speech Perception

Perceiving and remembering speech depend on multifractal nonlinearity in movements producing and exploring speech

Perceiving and remembering speech depend on multifractal nonlinearity in movements producing and exploring speech

Contact Info

Product

Resources

About