Data Collection for Dialogue System: A Startup Perspective

Kang, Yiping; Kummerfeld, Jonathan K.; Tang, Lingjia; Mars, Jason

doi:10.18653/v1/n18-3005

Cited by 26 publications

(39 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We build on prior work employing online crowd workers to create data by paraphrasing. In particular, we refine the idea of iteratively asking for paraphrases, where each round prompts workers with sentences from the previous round, leading to more diverse data (Negri et al, 2012;Jiang et al, 2017;Kang et al, 2018). We also apply the idea of a multi-stage process, in which a second set of workers check paraphrases to ensure they are correct (Buzek et al, 2010;Burrows et al, 2013;Coucke et al, 2018).…”

Section: Data Collectionmentioning

confidence: 99%

“…To demonstrate this idea, we developed a novel crowdsourcing pipeline for data collection. Following prior work in crowdsourcing for dialog (Kang et al, 2018;Jiang et al, 2017), we ask crowd workers to write paraphrases of seed sentences with known intents and slot values. This provides linguistic diversity in our data in a way that is easily explained to workers.…”

Section: Application: Uniqueness-driven Data Collectionmentioning

confidence: 99%

“…|n-grams a ∩ n-grams b | |n-grams a ∪ n-grams b | D(a, b)) Figure 5: Metrics for diversity and coverage from Kang et al (2018). X and Y are sets of utterances labeled with intents from I, and X i is the data in X for intent i.…”

Section: Tasksmentioning

confidence: 99%

“…The distance metric for comparing a pair of utterances is based on the Jaccard Index over n-grams. We follow the work from Kang et al (2018) and set N , the max n-gram length, to 3.…”

Section: Tasksmentioning

confidence: 99%

“…Intrinsic We measure the diversity and coverage of each dataset using the metrics introduced in (Kang et al, 2018) and shown in Figure 5.…”

Section: Evaluation Metricsmentioning

confidence: 99%

See 4 more Smart Citations

Outlier Detection for Improved Data Quality and Diversity in Dialog Systems

Larson¹,

Mahendran²,

Lee

et al. 2019

Proceedings of the 2019 Conference of the North

Self Cite

View full text Add to dashboard Cite

In a corpus of data, outliers are either errors: mistakes in the data that are counterproductive, or are unique: informative samples that improve model robustness. Identifying outliers can lead to better datasets by (1) removing noise in datasets and (2) guiding collection of additional data to fill gaps. However, the problem of detecting both outlier types has received relatively little attention in NLP, particularly for dialog systems. We introduce a simple and effective technique for detecting both erroneous and unique samples in a corpus of short texts using neural sentence embeddings combined with distance-based outlier detection. We also present a novel data collection pipeline built atop our detection technique to automatically and iteratively mine unique data samples while discarding erroneous samples. Experiments show that our outlier detection technique is effective at finding errors while our data collection pipeline yields highly diverse corpora that in turn produce more robust intent classification and slot-filling models.

show abstract

Section: Data Collectionmentioning

confidence: 99%

Section: Application: Uniqueness-driven Data Collectionmentioning

confidence: 99%

Section: Tasksmentioning

confidence: 99%

“…The distance metric for comparing a pair of utterances is based on the Jaccard Index over n-grams. We follow the work from Kang et al (2018) and set N , the max n-gram length, to 3.…”

Section: Tasksmentioning

confidence: 99%

“…Intrinsic We measure the diversity and coverage of each dataset using the metrics introduced in (Kang et al, 2018) and shown in Figure 5.…”

Section: Evaluation Metricsmentioning

confidence: 99%

See 3 more Smart Citations

Outlier Detection for Improved Data Quality and Diversity in Dialog Systems

Larson¹,

Mahendran²,

Lee

et al. 2019

Proceedings of the 2019 Conference of the North

Self Cite

View full text Add to dashboard Cite

show abstract

Automated Paraphrase Generation with Over-Generation and Pruning Services

Berro

Báez

Benatallah

et al. 2021

Service-Oriented Computing

View full text Add to dashboard Cite

Conversational services are emerging as a new paradigm for accessing information by simply uttering questions in natural language, posing a whole new set of challenges to the design and engineering of information systems. Training conversational services to deal with the nuances of natural language often requires collecting a high-quality and diverse set of training samples (i.e., paraphrases). Traditional approaches such as hiring an expert or crowdsourcing involve data collection processes that are often costly and time-consuming. Automated paraphrase generation is a promising cost-effective and scalable approach to generating training samples. Current automatic techniques, however, tend to specialise in specific types of lexical or syntactic variations. As a result, generated paraphrases may not perform well in relevant quality aspects such as diversity and semantic relatedness. In this paper, we follow an approach inspired by services integration to address these issues and generate paraphrases in English that are semantically relevant and diverse. We propose an extensible and reusable pipeline that combines automatic paraphrasing techniques in a two-step process that first focus on i) leveraging the strengths of multiple techniques to generate the most diverse (and possibly noisy) set of paraphrases, to then ii) address common quality issues in a separate step. Through empirical evaluations we show the benefits of the two-step process design and of combining techniques for more balancing relevance and diversity.

show abstract