Major–Minor Long Short-Term Memory for Word-Level Language Model

Abstract

Language model plays an important role in natural language processing (NLP) systems like machine translation, speech recognition, learning token embeddings, natural language generation and text classification. Recently, the multi-layer Long Short-Term Memory (LSTM) models have been demonstrated to achieve promising performance on word-level language modeling. For each LSTM layer, larger hidden size usually means more diverse semantic features, which enables the language model to perform better. However, we have observed that when a certain LSTM layer reaches a sufficiently large scale, the promotion of overall effect will slow down as its hidden size increases. In this paper, we analyze that an important factor leading to this phenomenon is the high correlation between the newly extended hidden states and original hidden states, which hinders diverse feature expression of the LSTM. As a result, when the scale is large enough, simply lengthening the LSTM hidden states will cost tremendous extra parameters but has little effect. We propose a simple yet effective improvement on each LSTM layer consisting of a large-scale Major LSTM and a smallscale Minor LSTM to break the high correlation between the two parts of hidden states, which we call Major-Minor LSTMs (MMLSTMs). In experiments, we demonstrate the language model with MMLSTMs surpasses the existing state-of-the-art model on Penn Treebank (PTB) and WikiText-2 (WT2) datasets, and outperforms the baseline by 3.3 points in perplexity on WikiText-103 dataset without increasing model parameter counts.

FastAdaBelief: Improving Convergence Rate for Belief-Based Adaptive Optimizers by Exploiting Strong Convexity

“…In this group of experiments, we use a classic recurrent network (i.e., LSTM) and an open dataset (i.e., Penn Treebank). Same as the previous works [22], [23], [32], [33], we take the perplexity to measure the performance of all the comparison algorithms.…”

“…Since β 1t ≤ β 1 and from Equation(32), we can further attain the following bound for E 2 :E 2 ≤ α 2 (1 − β1) n i=1 T j=1 T −j k=1 β 1(T −k+1) g 2 j,i T T j=1 (1 − β2j) T −j k=1 β 2(T −k+1) g 2 j,i + δ ≤ α 2 (1 − β1) + ζδ, and ω 0 = ζδ, we obtain Ψ = ω j − ω j−1 ω j . (In addition, for any a ≥ b > 0, the inequality 1 + x ≤ e x implies that…”

A NN-based attention model for rope vibration prediction and channel sensitivity analysis

“…It is easy to find the Model-SLD is designed based on Model 2 with the addition of One-Dimensional Convolution Network 30 and LSTM (Long and Short Time Network). 31,32…”

FastAdaBelief: Improving Convergence Rate for Belief-based Adaptive Optimizers by Exploiting Strong Convexity

“…In this group of experiments, we use a classic recurrent network (i.e., LSTM) and an open dataset (i.e., Penn Treebank). Same as the previous works [16], [17], [22], [23], we use perplexity to measure the performance of all the comparison algorithms. Note that the lower perplexity is better.…”

“…In addition, for term E 1 of Equation ( 21), and from Equation ( 6), we can obtain (22) where i ∈ {1, . .…”