A theory of word frequencies and its application to dialogue move recognition

Abstract

Dialogue move recognition is taken as being representative of a class of spoken language applications where inference about high level semantic meaning is required from lower level acoustic, phonetic or word based features. Topic identication is another such application. In the particular case of inference from words, the multinomial distribution is shown to be inadequate for modelling word frequencies, and the multivariate Poisson is a more reasonable choice. Zipf's law is used to model a prior distribution. This more rigorous mathematical formulation is shown to improve dialogue move classication both subjectively and quantitatively.

Surface and Contextual Linguistic Cues in Dialog Act Classification: A Cognitive Science View

“…Word sequences are the most frequently used cue (e.g., Dai et al, 2020;Grau et al, 2004;Kalchbrenner & Blunsom, 2013;Stolcke et al, 2000), likely because many dialog act classification frameworks heavily rely on lexical information (Duran & Battle, 2018;Jurafsky et al, 1997;Louwerse & Crossley, 2006). In frequency-based and machine learning approaches, word sequences have typically been encoded using n-grams (e.g., Garner et al, 1996;Grau et al, 2004;Ribeiro et al, 2015;Louwerse & Crossley, 2006). Deep learning methods have typically used recurrent neural networks to encode such sequences (e.g., Dai et al, 2020;Ji et al, 2016;Tran et al, 2017b;Zhao & Kawahara, 2019).…”

“…All five dialog act classification studies using a frequency‐based approach that we reviewed (Table 1, Studies 1–5) made use of word sequence statistics in the form of word n ‐grams to represent utterances. While one of the reviewed studies did not make use of contextual cues (Webb et al., 2005), another study used information on who is the speaker as contextual cues (Louwerse & Crossley, 2006), and yet other studies represented context through incorporating information on the dialog acts of the previous utterances (Garner et al., 1996; Ji & Bilmes, 2005; Stolcke et al., 2000).…”