Personalizing universal recurrent neural network language model with user characteristic features by social network crowdsourcing

Abstract

With the popularity of mobile devices, personalized speech recognizer becomes more realizable today and highly attractive. Each mobile device is primarily used by a single user, so it's possible to have a personalized recognizer well matching to the characteristics of individual user. Although acoustic model personalization has been investigated for decades, much less work have been reported on personalizing language model, probably because of the difficulties in collecting enough personalized corpora. Previous work used the corpora collected from social networks to solve the problem, but constructing a personalized model for each user is troublesome. In this paper, we propose a universal recurrent neural network language model with user characteristic features, so all users share the same model, except each with different user characteristic features. These user characteristic features can be obtained by crowdsouring over social networks, which include huge quantity of texts posted by users with known friend relationships, who may share some subject topics and wording patterns. The preliminary experiments on Facebook corpus showed that this proposed approach not only drastically reduced the model perplexity, but offered very good improvement in recognition accuracy in n-best rescoring tests. This approach also mitigated the data sparseness problem for personalized language models.

Deep Learning for Mobile Crowdsourcing Techniques, Methods, and Challenges: A Survey

“…It combines with convolutional neural network in deep learning to filter redundant photo information. In order to capture the characteristics of users, Tseng et al [51] proposed a speech recognition model named RNNLM, which uses RNN to capture the features of different users. Toman et al [52] used the basic deep learning model DNN for crowdsourcing voice data collection, through data selection and enhancement of a large number of voices.…”

“…Hamrouni et al [50] Presenting photo-based MCS framework. Tseng et al [51] Presenting a speech recognition model RNNLM. Toman et al [52] Using the basic deep learning model DNN for crowdsourcing voice data collection.…”

Proceedings of the First Workshop on Natural Language Interfaces

“…Language modeling is a critical component for many NLIs, and personalization is a natural direction to improve these interfaces. Several published works have explored personalization of language models using historical search queries (Jaech and Ostendorf, 2018), features garnered from social graphs (Wen et al, 2012;Tseng et al, 2015;Lee et al, 2016), and transfer learning techniques (Yoon et al, 2017). Other work has explored using profile information (location, name, etc.)…”

Gmail Smart Compose

“…The Smart Compose model is extended for personalization, as detailed in Section 5. As an important feature, personalization has been considered as a specific language model adaptation techniques in various applications, see for example [21,22,34,47,50]. In terms of the personalized model architecture, perhaps most close to our approach is Chen et al [11], which is also a linear interpolation between an n-gram and recurrent neural network language model.…”