Comparative Error Analysis of Dialog State Tracking

Abstract

A primary motivation of the Dialog State Tracking Challenge (DSTC) is to allow for direct comparisons between alternative approaches to dialog state tracking. While results from DSTC 1 mention performance limitations, an examination of the errors made by dialog state trackers was not discussed in depth. For the new challenge, DSTC 2, this paper describes several techniques for examining the errors made by the dialog state trackers in order to refine our understanding of the limitations of various approaches to the tracking process. The results indicate that no one approach is universally superior, and that different approaches yield different error type distributions. Furthermore, the results show that a pairwise comparative analysis of tracker performance is a useful tool for identifying dialogs where differential behavior is observed. These dialogs can provide a data source for a more careful analysis of the source of errors.

F-Measure Optimisation and Label Regularisation for Energy-Based Neural Dialogue State Tracking Models

“…To our knowledge only one example of a comparative error analysis of dialogue state trackers was given by Smith [12]. In that analysis the author reports the error distributions of tracking models over three error types of possible deviations from the true joint goal for every turn in the dialogue.…”

The Dialog State Tracking Challenge Series: A Review

“…As pointed out by Smith (2014), it is important to examine the types of errors made by a tracker in order to make improvements. To do this, at each turn, we compare the top dialog state output by each tracker with the true dialog state, and examine each slot.…”

“…Description based on survey collected from participants. (Kim and Banchs, 2014) Linear CRF team3 entry0 (Smith, 2014) Discourse rules + dialog act bigrams team4 entry2 (Henderson et al, 2014d) Recurrent neural network team6 entry2…”

“…This has had unforeseen benefits: first, the DSTC data now forms a sort of benchmark for the field, with groups continuing to report results on it after the challenge proper (Lee, 2013;Ma and Fosler-Lussier, 2014b;Zilka and Jurčíček, 2015;Fix and Frezza-Buet, 2015). In addition, the DSTC1-3 corpora have been used to examine which state tracking evaluation metrics correlate with dialog success (Lee, 2014), perform detailed error analyses of state trackers (Smith, 2014), and for dialog act classification and SLU experimentation (Ma and Fosler-Lussier, 2014a;Ferreira et al, 2015). We encourage future challenges to continue this tradition.…”

Hybrid Dialog State Tracker with ASR Features

“…One such example can be "asian food" which appears 16 times in the training data as a part of the best ASR hypothesis while 13 times it really informs about "asian oriental" ontology value. Measurements on dstc2 dev have shown Williams (2014) .739 .721 Henderson et al (2014b) .737 .406 Knowledge-based tracker (Kadlec et al, 2014) .737 .429 √ et al (2014) .735 .433 Smith (2014) .729 .452 Lee et al (2014) .726 .427 YARBUS (Fix and Frezza-buet, 2015) . Table 1: Joint slot tracking accuracy and L2 (denotes the squared L2 norm between the estimated belief distribution and correct distribution) for various systems reported in the literature.…”