Overview of the Eighth Dialog System Technology Challenge: DSTC8

Kim, Seokhwan; Galley, Michel; Gunasekara, Chulaka; Lee, Sung‐Jin; Atkinson, Adam; Peng, Baolin; Schulz, Hannes; Gao, Jianfeng; Li, Jinchao; Adada, Mahmoud; Huang, Minlie; Lastras, Luis A.; Kummerfeld, Jonathan K.; Lasecki, Walter S.; Hori, Chiori; Cherian, Anoop; Marks, Tim K.; Rastogi, Abhinav; Zang, Xiaoxue; Sunkara, Srinivas; Gupta, Raghav

doi:10.1109/taslp.2021.3078368

Cited by 10 publications

(5 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This task is a continuation of last year at DSTC8 [16]. Participants will develop an end-to-end task-oriented dialog system that takes natural language as input and generates natural language response as output in the travel planning setting.…”

Section: A End-to-end Task-oriented Dialog Taskmentioning

confidence: 99%

Overview of the Ninth Dialog System Technology Challenge: DSTC9

Gunasekara¹,

Kim²,

D’Haro³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

This paper introduces the Ninth Dialog System Technology Challenge (DSTC-9). This edition of the DSTC focuses on applying end-to-end dialog technologies for four distinct tasks in dialog systems, namely, 1. Task-oriented dialog Modeling with unstructured knowledge access, 2. Multi-domain task-oriented dialog, 3. Interactive evaluation of dialog, and 4. Situated interactive multi-modal dialog. This paper describes the task definition, provided datasets, baselines and evaluation set-up for each track. We also summarize the results of the submitted systems to highlight the overall trends of the state-of-the-art technologies for the tasks.

show abstract

Section: A End-to-end Task-oriented Dialog Taskmentioning

confidence: 99%

Overview of the Ninth Dialog System Technology Challenge: DSTC9

Gunasekara¹,

Kim²,

D’Haro³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We base the new Audio-Visual Scene-Aware Dialog (AVSD) task for DSTC10 on the AVSD dataset from DSTC7-8 [1,2]. For the AVSD data, we collected text-based dialogs on short videos from the popular Charades dataset [3], which consists of untrimmed and multi-action videos (each video also has an audio track) and comes with human-generated descriptions of the scene.…”

Section: Audio-visual Scene-aware Dialog Data Setmentioning

confidence: 99%

“…To encourage development of dialog system technologies that enable an agent to discuss audio-visual scenes with humans, we held two challenges on audio-visual Scene-Aware Dialog (AVSD) at DSTC7 and DSTC8 [1,2] using a dataset we collected based on the videos from the Charades dataset [3]. The AVSD task we defined and dataset we prepared were the first attempt to promote the combination of audio-visual question-answering systems and conversation systems into a single framework [4,5].…”

Section: Introductionmentioning

confidence: 99%

Audio-Visual Scene-Aware Dialog and Reasoning using Audio-Visual Transformers with Joint Student-Teacher Learning

Shah¹,

Geng²,

Gao³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In previous work, we have proposed the Audio-Visual Scene-Aware Dialog (AVSD) task, collected an AVSD dataset, developed AVSD technologies, and hosted an AVSD challenge track at both the 7th and 8th Dialog System Technology Challenges (DSTC7, DSTC8). In these challenges, the best-performing systems relied heavily on human-generated descriptions of the video content, which were available in the datasets but would be unavailable in real-world applications. To promote further advancements for real-world applications, we proposed a third AVSD challenge, at DSTC10, with two modifications: 1) the human-created description is unavailable at inference time, and 2) systems must demonstrate temporal reasoning by finding evidence from the video to support each answer. This paper introduces the new task that includes temporal reasoning and our new extension of the AVSD dataset for DSTC10, for which we collected human-generated temporal reasoning data. We also introduce a baseline system built using an AV-transformer, which we released along with the new dataset. Finally, this paper introduces a new system that extends our baseline system with attentional multimodal fusion, joint student-teacher learning (JSTL), and model combination techniques, achieving state-of-the-art performances on the AVSD datasets for DSTC7, DSTC8, and DSTC10. We also propose two temporal reasoning methods for AVSD: one attention-based, and one based on a time-domain region proposal network.

show abstract

“…

This paper introduces one of our group's work on the Dialog System Technology Challenges 8 (DSTC8) (Kim et al 2019), the SPPD system for Schema Guided dialogue state tracking challenge. This challenge, named as Track 4 in DSTC8, provides a brand new and challenging dataset for developing scalable multi-domain dialogue state tracking algorithms for real world dialogue systems.

…”

mentioning

confidence: 99%

The SPPD System for Schema Guided Dialogue State Tracking Challenge

Li,

Xiong,

Cao

2020

Preprint

View full text Add to dashboard Cite

This paper introduces one of our group's work on the Dialog System Technology Challenges 8 (DSTC8) (Kim et al. 2019), the SPPD system for Schema Guided dialogue state tracking challenge. This challenge, named as Track 4 in DSTC8, provides a brand new and challenging dataset for developing scalable multi-domain dialogue state tracking algorithms for real world dialogue systems. We propose a zero-shot dialogue state tracking system for this task. The key components of the system is a number of BERT based zero-shot NLU models that can effectively capture semantic relations between natural language descriptions of services' schemas and utterances from dialogue turns. We also propose some strategies to make the system better to exploit information from longer dialogue history and to overcome the slot carryover problem for multi-domain dialogues. The experimental results show that the proposed system achieves a significant improvement compared with the baseline system. Recently, the Dialogue State Tracking Challenges (DSTC) (Williams et al. 2013;Henderson, Thomson, and Williams 2014a;Henderson, Thomson, and Williams 2014b;Kim et al. 2017;Kim et al. 2016) provide shared benchmarks to the research community and a variety of state tracking models are proposed. There are rule-based models (Wang and Lemon 2013), statistical models such as generative models (Thomson and Young 2010) and discriminative models (Lee 2013).

show abstract

Overview of the Eighth Dialog System Technology Challenge: DSTC8

Cited by 10 publications

References 32 publications

Overview of the Ninth Dialog System Technology Challenge: DSTC9

Overview of the Ninth Dialog System Technology Challenge: DSTC9

Audio-Visual Scene-Aware Dialog and Reasoning using Audio-Visual Transformers with Joint Student-Teacher Learning

The SPPD System for Schema Guided Dialogue State Tracking Challenge

Contact Info

Product

Resources

About