Reusing Weights in Subword-Aware Neural Language Models

Abstract

We propose several ways of reusing subword embeddings and other weights in subwordaware neural language models. The proposed techniques do not benefit a competitive character-aware model, but some of them improve the performance of syllable-and morpheme-aware models while showing significant reductions in model sizes. We discover a simple hands-on principle: in a multilayer input embedding model, layers should be tied consecutively bottom-up if reused at output. Our best morpheme-aware model with properly reused weights beats the competitive word-level model by a large margin across multiple languages and has 20%-87% fewer parameters.

A Simple and Effective Method for Injecting Word-Level Information into Character-Aware Neural Language Models

“…In addition to the above datasets, we also set up 6 common language modeling datasets: English Penn Treebank (PTB) (Marcus et al, 1993) and 5 non-English datasets with rich morphology from the 2013 ACL Workshop on Machine Translation 5 , which have been commonly used for evaluating character-aware NLMs (Botha and Blunsom, 2014;Kim et al, 2016;Bojanowski et al, 2017;Assylbekov and Takhanov, 2018). Since some of previous work has tested their model on PTB, we also included PTB in our experiment.…”

“…Bojanowski et al (2017) trained the word embeddings through skip-gram models with subword-level information, and used these word embeddings to initialize the lookup table of word embeddings of a word-level language model. Assylbekov and Takhanov (2018) focused on reusing embeddings and weights in a characteraware language model. The input of their model is also the sum of the morpheme embeddings of the word.…”

“…Many work have attempted to improve characteraware NLMs in recent years. For example, Assylbekov and Takhanov (2018) proposed several ways of reusing weights in character-aware NLMs. Gerz et al (2018) achieved an improved result on 50 typologically diverse languages by injecting subword-level information into word vectors at the softmax.…”

“…The language models used in the previous work are improved at different aspects, and most of them are also based on standard LSTM, like ours. Botha PTB CS DE ES FR RU MLBL (Botha and Blunsom, 2014) -465 296 200 225 304 MorphSum (Kim et al, 2016) -398 263 196 271 CharCNN (Kim et al, 2016) 78.9 371 239 165 184 261 SkipGram initialization (Bojanowski et al, 2017) -312 206 145 159 206 MorphSum+RE+RW (Assylbekov and Takhanov, 2018) and Blunsom (2014) used the morphological logbilinear (MLBL) model, which takes into account morpheme information. Kim et al (2016) used CNN as their character encoder, and also trained an LSTM language model, where the input representation of a word is the sum of the morpheme embeddings of the word.…”

“…125 538 105 182 316 430 1339 1474 536 1116 260 1103 659 1728 Char-BiLSTM-cat-Word-LSTM 133 565 105 183 314 432 1239 1360 504 1052 245 993 614 1602 Char-BiLSTM-cat-Word-LSTM-Word (g = 0.5, n = 1) 122 541 103 180 305 426 1158 1316 530 1031 241 1012 607 1561 Char-BiLSTM-avg-Word-LSTM 133 609 110 177 307 461 1181 1340 478 963 225 996 611 (Kim et al, 2016) 78.9 371 239 165 184 261 SkipGram initialization (Bojanowski et al, 2017) -312 206 145 159 206 MorphSum+RE+RW (Assylbekov and Takhanov, 2018)…”