Improving Out-of-Scope Detection in Intent Classification by Using Embeddings of the Word Graph Space of the Classes

Abstract

This paper explores how intent classification can be improved by representing the class labels not as a discrete set of symbols but as a space where the word graphs associated to each class are mapped using typical graph embedding techniques. The approach, inspired by a previous algorithm used for an inverse dictionary task, allows the classification algorithm to take in account inter-class similarities provided by the repeated occurrence of some words in the training examples of the different classes. The classification is carried out by mapping text embeddings to the word graph embeddings of the classes. Focusing solely on improving the representation of the class label set, we show in experiments conducted in both private and public intent classification datasets, that better detection of out-of-scope examples (OOS) is achieved and, as a consequence, that the overall accuracy of intent classification is also improved. In particular, using the recently-released Larson dataset, an error of about 9.9% has been achieved for OOS detection, beating the previous state-of-the-art result by more than 31 percentage points.

Using Meta-Knowledge Mined from Identifiers to Improve Intent Recognition in Conversational Systems

“…We present now a formal description of the methodology employed in this work which takes advantage of the intent proto-taxonomies using a neurosymbolic approach. It expands some previous work which focused on the use of keywords as the source of symbolic information (Cavalin et al, 2020). .…”

“…However, the dataset explored in that work is very limited. Recent work has also demonstrated that intent recognition can be improved by enhancing class representations with "keywords" which are extracted from exemplar utterances considering their most common words (Cavalin et al, 2020).…”

“…We aim not only to provide methods more aligned to everyday class definition practices of developers but also to improve the accuracy of the ML methods. Inspired by a reverse dictionary algorithm (Kartsaklis et al, 2018) and previous work on keyword-based classification (Cavalin et al, 2020), we propose three neuro-symbolic algorithms which combine taxonomic descriptions of classes with traditional exemplar-based supervised learning. We show that those novel algorithms are able to decrease error rates for a significant number of datasets, particularly in the difficult task of detecting OOS cases in real, professional chatbots.…”

Out-of-Scope Intent Detection on A Knowledge-Based Chatbot

“…A neural sentence embedding approach representing sentences in a low-dimensional vector space while emphasizing IS characteristics that differentiate from OOS cases was employed in [13]. The OOS classification could also be accomplished by mapping text embeddings of IS data to the groups' word graph space [14]. Furthermore, Kullback-Leibler (KL) divergence is applied in [15] to gather information between the intent distributions of consecutive words.…”

“…OOS classification methods are categorized into two approaches, few-shot learning (FSL) and zeroshot learning (ZSL). While FSL tries to create machine learning models using a small amount of OOS training data, such as in this work, Google DialogFlow, and [15], ZSL relies solely on IS training data to detect an OOS like in [13,14]. However, both FSL and ZSL still require labeled IS datasets.…”