Intent Mining from past conversations for Conversational Agent

Abstract

Conversational systems are of primary interest in the AI community. Organizations are increasingly using chatbot to provide round-the-clock support and to increase customer engagement. Many commercial bot building frameworks follow a standard approach that requires one to build and train an intent model to recognize user input. These frameworks require a collection of user utterances and corresponding intent to train an intent model. Collecting a substantial coverage of training data is a bottleneck in the bot building process. In cases where past conversation data is available, the cost of labeling hundreds of utterances with intent labels is time-consuming and laborious. In this paper, we present an intent discovery framework that can mine a vast amount of conversational logs and to generate labeled data sets for training intent models. We have introduced an extension to the DBSCAN (Ester et al., 1996) algorithm and presented a density-based clustering algorithm ITER-DBSCAN for unbalanced data clustering. Empirical evaluation on one conversation dataset, six intent dataset, and one short text clustering dataset show the effectiveness of our hypothesis. We release the datasets and code for future evaluation at https://github.com/ajaychatterjee/IntentMining . 1

New Intent Discovery with Pre-training and Contrastive Learning

“…Perkins and Yang (2019) considered the context of an utterance in a conversation. Chatterjee and Sengupta (2020) proposed to improve density-based models. Some recent works (Haponchyk et al, 2018;Haponchyk and Moschitti, 2021) studied supervised clustering algorithms for intent labeling, yet it can not handle new intents.…”

Confidence Calibration for Intent Detection via Hyperspherical Space and Rebalanced Accuracy-Uncertainty Loss

“…Intent is the sematic purpose of a query, which is generated by users (Xu and Sarikaya 2013;Wang, Tang, and He 2018). As a matter of fact, the essence of intent detection is text classification (Brenes, Gayo-Avello, and Pérez-González 2009;Mehri, Eric, and Hakkani-Tur 2020;Chatterjee and Sengupta 2020). After training on the dataset with ground-truth labels, the model attempts to predict the intent of query within the existing intent set.…”

Open Intent Discovery through Unsupervised Semantic Clustering and Dependency Parsing

“…The two encoders are updated iteratively using the cluster assignment obtained from the alternative view to encourage them to yield similar cluster assignments for the same user queries. Along the direction of unsupervised clustering approaches, Chatterjee and Sengupta [28] present an intent discovery framework for conversation data, which consists of dialog act classification, density-based clustering, manual annotation of clusters and propagation of intent labels. In comparison with previous approaches, this work differs in three major aspects: (1) we adopt pre-trained language models for utterance representation; (2) we incorporate K-means with a penalty term to learn the optimal number of clusters; and (3) we generate the cluster labels automatically using a dependency parser.…”

“…Since the SNIPS dataset has ground-truth intent labels, we adopt precision (P), recall (R) and F1 to evaluate the proposed intent discovery framework. Following [28], we also use the Normalized Mutual Information (NMI [35]) and Adjusted Rand Index (ARI [36]) as the metrics for evaluating clustering performance. NMI normalizes the mutual information between a predicted clustering and the true clustering, and ranges from 0 to 1; while ARI computes the similarity between two clusterings by considering all pairs of examples and counting the proportion of pairs that are assigned to the same or different clusters.…”