Parsing Speech: a Neural Approach to Integrating Lexical and Acoustic-Prosodic Information

Abstract

In conversational speech, the acoustic signal provides cues that help listeners disambiguate difficult parses. For automatically parsing spoken utterances, we introduce a model that integrates transcribed text and acoustic-prosodic features using a convolutional neural network over energy and pitch trajectories coupled with an attention-based recurrent neural network that accepts text and prosodic features. We find that different types of acoustic-prosodic features are individually helpful, and together give statistically significant improvements in parse and disfluency detection F1 scores over a strong text-only baseline. For this study with known sentence boundaries, error analyses show that the main benefit of acousticprosodic features is in sentences with disfluencies, attachment decisions are most improved, and transcription errors obscure gains from prosody.

Giving Attention to the Unexpected: Using Prosody Innovations in Disfluency Detection

“…Fundamental frequency (F0) and Energy (E). Similar to Tran et al (2018), we use three F0 features and three energy features. The three F0 features include normalized cross correlation function (NCCF), log-pitch weighted by probability of voicing (POV), and the estimated delta of log pitch.…”

Disfluencies and Human Speech Transcription Errors

“…This work was motivated in part by a prior study showing that transcription errors impact findings related to the usefulness of prosodic features in parsing [18], i.e., a significant fraction of the cases where prosody seems to hurt parsing are associated with transcription errors. The availability of the new disfluency annotations will make it possible to explore this question for disfluencies.…”

An Empirical Analysis of the Correlation of Syntax and Prosody

“…Our statistical analysis is hence conservative and points out only the most direct and most relevant correlations. Feature combinations instead of simple flat representations have lead to a break-through in parsing [9]. Hence, we expect many more and more complex interplays in the syntax-prosody interface to be found in future work building on more complex notions of prosodic and syntactic features.…”