A Conditional Generative Matching Model for Multi-lingual Reply Suggestion

Abstract

We study the problem of multilingual automated reply suggestions (RS) model serving many languages simultaneously. Multilingual models are often challenged by model capacity and severe data distribution skew across languages. While prior works largely focus on monolingual models, we propose Conditional Generative Matching models (CGM), optimized within a Variational Autoencoder framework to address challenges arising from multilingual RS. CGM does so with expressive message conditional priors, mixture densities to enhance multi-lingual data representation, latent alignment for language discrimination, and effective variational optimization techniques for training multi-lingual RS. The enhancements result in performance that exceed competitive baselines in relevance (ROUGE score) by more than 10% on average, and 16% for low resource languages. CGM also shows remarkable improvements in diversity (80%) illustrating its expressiveness in representation of multi-lingual data.

End-to-End Autoregressive Retrieval via Bootstrapping for Smart Reply Systems

“…Smart reply The proprietary nature of data from email and chat applications has led several previous works to use publicly-available dialogue datasets (Zhang et al, 2021;Deb et al, 2021;Towle and Zhou, 2023) to benchmark SR methods, due to their analogous conversational nature. While early SR systems used generative models (Kannan et al, 2016), current production systems favour retrieval methods due to their greater controllability of outputs and superior latency (Deb et al, 2019).…”

“…We evaluate our method on the same weighted ROUGE ensemble as previous methods (Lin, 2004;Deb et al, 2019Deb et al, , 2021, which is known to correlate well with click-through rate (Zhang et al, 2021):…”

“…Several prior works explore solutions to this problem such as removing near duplicates, penalising inter-reply similarity (Deb et al, 2019), clustering by intent (Henderson et al, 2017;Weng et al, 2019), learning latent variables (Deb et al, 2019(Deb et al, , 2021, or model-based simulation (Towle and Zhou, 2023). However, these methods share a common design choice (Figure 1A): (1) a retrievalbased Matching model, which has learned a shared embedding space between messages and replies, returns a shortlist of top scoring replies; (2) this shortlist is refined through some diversification procedure to obtain the final reply set.…”