A Re-Ranker Scheme For Integrating Large Scale NLU Models

Abstract

Large scale Natural Language Understanding (NLU) systems are typically trained on large quantities of data, requiring a fast and scalable training strategy. A typical design for NLU systems consists of domain-level NLU modules (domain classifier, intent classifier and named entity recognizer). Hypotheses (NLU interpretations consisting of various intent+slot combinations) from these domain specific modules are typically aggregated with another downstream component. The re-ranker integrates outputs from domain-level recognizers, returning a scored list of cross domain hypotheses. An ideal reranker will exhibit the following two properties: (a) it should prefer the most relevant hypothesis for the given input as the top hypothesis and, (b) the interpretation scores corresponding to each hypothesis produced by the re-ranker should be calibrated. Calibration allows the final NLU interpretation score to be comparable across domains. We propose a novel re-ranker strategy that addresses these aspects, while also maintaining domain specific modularity. We design optimization loss functions for such a modularized re-ranker and present results on decreasing the top hypothesis error rate as well as maintaining the model calibration. We also experiment with an extension involving training the domain specific re-rankers on datasets curated independently by each domain to allow further asynchronization.

Fast Intent Classification for Spoken Language Understanding Systems

“…As the next steps, we aim at extending the BranchyNet scheme to the entire stack of SLU models [2]. The BranchyNet scheme can also be combined with other modeling schemes such as model distillation and compression.…”

“…Spoken Language Understanding(SLU) systems are core components of voice agents such as Apple's Siri, Amazon Alexa and Google Assistant and can be designed in one of the several ways, such as an end to end modeling scheme [1], or a collection of task specific classifiers [2,3]. For a complex SLU system, the machine learning architecture can be computationally expensive, posing a challenge for applications such as On-Device-SLU.…”

“…For instance, a request such as "Stop" is arguably simpler to process for an SLU system in comparison to a more complex request such as "find me the closest open restaurant at 8 PM and reserve me a spot". To make use of this specificity in terms of computational load, we use the BranchyNet modeling scheme for an intent classification model as is used in the SLU system described in [2]. The BranchyNet scheme allows for decision making at various depths in a deep learning model, thereby naturally fitting our use case.…”

Efficient Semi-Supervised Learning for Natural Language Understanding by Optimizing Diversity

“…We used n-grams extracted from training data as features for the models. A detailed description of our NLU system can be found in [25].…”

One-vs-All Models for Asynchronous Training: An Empirical Analysis

“…We conduct experiments on Natural Language Understanding (NLU) tasks, specifically a domain classification task and an SLU task to obtain domain-intent-entity combinations [10]. Although both multi-class [11,12] and OVA designs [13] exist for such tasks, a comparison between the two systems with asynchrony capabilities added in the OVA setup has not been explored.…”