Likelihood Ratios and Generative Classifiers for Unsupervised Out-of-Domain Detection in Task Oriented Dialog

Abstract

The task of identifying out-of-domain (OOD) input examples directly at test-time has seen renewed interest recently due to increased real world deployment of models. In this work, we focus on OOD detection for natural language sentence inputs to task-based dialog systems. Our findings are three-fold:First, we curate and release ROSTD (Real Out-of-Domain Sentences From Task-oriented Dialog) - a dataset of 4K OOD examples for the publicly available dataset from (Schuster et al. 2019). In contrast to existing settings which synthesize OOD examples by holding out a subset of classes, our examples were authored by annotators with apriori instructions to be out-of-domain with respect to the sentences in an existing dataset.Second, we explore likelihood ratio based approaches as an alternative to currently prevalent paradigms. Specifically, we reformulate and apply these approaches to natural language inputs. We find that they match or outperform the latter on all datasets, with larger improvements on non-artificial OOD benchmarks such as our dataset. Our ablations validate that specifically using likelihood ratios rather than plain likelihood is necessary to discriminate well between OOD and in-domain data.Third, we propose learning a generative classifier and computing a marginal likelihood (ratio) for OOD detection. This allows us to use a principled likelihood while at the same time exploiting training-time labels. We find that this approach outperforms both simple likelihood (ratio) based and other prior approaches. We are hitherto the first to investigate the use of generative classifiers for OOD detection at test-time.

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers

“…We selected two baselines to evaluate improvements of our proposed OodGAN. Our baselines for the ROSTD dataset is our implementation of Zheng et al (2020) and the work of Gangal et al (2019). The baseline for the OSQ dataset is our implementation of Zheng et al (2020).…”

“…They propose three baseline approaches for OOD detection that rely on OOD training data. Gangal et al (2019) created a ROSTD dataset and explored likelihood ratio based approaches. Lee and Shalyminov (2019) proposed an OOD detection method that does not require OOD data by utilizing counterfeit OOD turns in the context of a dialog.…”

Boosting Low-Resource Intent Detection with in-Scope Prototypical Networks

“…We conduct experiments on two public benchmark datasets after low-resource settings, including CLINC [15], ROSTD [9]. CLINC and ROSTD are for evaluating the performance of intent classification systems in the presence of OOD samples.…”

“…3) LSTM-AutoEncoder [5] trains an autoencoder for OOD detection using only ID data. 4) LR-GC is proposed by [9] to use a likelihood ratio method based on generative classifiers to correct confounding background statistics. 5) ProtoNet [10] is trained only on the naive classification loss L naive .…”

“…Given the considerable amount of labeled training data, recent methods [5,6,7,8,9] have been proven to be promising in OOD detection. However, they struggle in the lowresource regime.…”