Benchmarking Commercial Intent Detection Services with Practice-Driven Evaluations

Abstract

Intent detection is a key component of modern goal-oriented dialog systems that accomplish a user task by predicting the intent of users' text input. There are three primary challenges in designing robust and accurate intent detection models. First, typical intent detection models require a large amount of labeled data to achieve high accuracy. Unfortunately, in practical scenarios it is more common to find small, unbalanced, and noisy datasets. Secondly, even with large training data, the intent detection models can see a different distribution of test data when being deployed in the real world, leading to poor accuracy. Finally, a practical intent detection model must be computationally efficient in both training and single query inference so that it can be used continuously and retrained frequently. We benchmark intent detection methods on a variety of datasets. Our results show that Watson Assistant's intent detection model outperforms other commercial solutions and is comparable to large pretrained language models while requiring only a fraction of computational resources and training data. Watson Assistant demonstrates a higher degree of robustness when the training and test distributions differ.

Improved Text Classification via Contrastive Adversarial Training

“…For the three intent classification datasets, in addition to the original evaluation data, we also evaluate on a difficult subset of each test set described in (Qi et al 2021). The difficult subsets are constructed by comparing the TF-IDF vector of each test example to that of the training examples for a given intent.…”

“…We also experimented with generating our own difficult subsets in a similar manner using BERT-based sentence encoders 5 , and compare each test example with the mean-pooling of the training examples for that intent. Result shows that the TF-IDF method yields a more challenging subset, thus we report results on the original subsets from Qi et al (2021). The evaluation metric for all intent classification datasets is accuracy.…”

“…5 The specific model used is "paraphrase-mpnet-base-v2" (Song et al 2020) On the three intent classification datasets, we follow the hyperparameter settings in Qi et al (2021). We use a batch size of 32 and fine-tune for 5 epochs, and search over lr ∈ {0.00003, 0.00004, 0.00005}.…”

Fast and Light-Weight Answer Text Retrieval in Dialogue Systems

“…Query-document pairs are concatenated and sent through Transformer-based encoders, an additional layer on top of the encoded representation is adopted to produce a relevance score of the document to the query, which is then used for ranking. Arora et al (2020) and Qi et al (2021) benchmark intent detection models on intent detection datasets such as CLINC150 (Larson et al, 2019) where sufficient training examples exist for each intent. On the other hand, our use case focuses on the scenarios where answer text is available but training examples are insufficient.…”

Proceedings of the Workshop on Multilingual Information Access (MIA)

“…We also create the fewshot version of these datasets to evaluate the models' performance on small datasets. Additionally, after observing the close accuracy results among the models, we follow Arora et al ( 2020) and Qi et al (2021) to create the TF*IDF and jaccard based difficult testing set to differentiate them better. 4 Overall, our benchmark generates about 1000 data points, including accuracy and training time in default, few-shot training, and difficult testing settings.…”