Structured Word Embedding for Low Memory Neural Network Language Model

Abstract

Neural network language model (NN LM), such as long short term memory (LSTM) LM, has been increasingly popular due to its promising performance. However, the model size of an uncompressed NN LM is still too large to be used in embedded or portable devices. The dominant part of memory consumption of NN LM is the word embedding matrix. Directly compressing the word embedding matrix usually leads to performance degradation. In this paper, a product quantization based structured embedding approach is proposed to significantly reduce memory consumption of word embeddings without hurting LM performance. Here, each word embedding vector is cut into partial embedding vectors which are then quantized separately. Word embedding matrix can then be represented by an index vector and a code-book tensor of the quantized partial embedding vectors. Experiments show that the proposed approach can achieve 10 to 20 times embedding parameter reduction rate with negligible performance loss.

Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decomposition

“…However, running these models on edge-devices, faces memory and latency issues due to limitations of the hardware. Thus, there has been considerable interest towards research in reducing the memory footprint and faster inference speed for these models (Sainath et al, 2013;Acharya et al, 2019;Shi and Yu, 2018;Jegou et al, 2010;Chen et al, 2018;Winata et al, 2019).…”

“…Recently, there has been considerable work on compressing word-embedding matrices (Sainath et al, 2013;Acharya et al, 2019;Shi and Yu, 2018;Jegou et al, 2010;Chen et al, 2018;Winata et al, 2019). These techniques have proven to perform atpar with the uncompressed models, but still suffer from a number of issues.…”

“…First, state-of-the-art embedding compression methods such as GroupReduce, Structured Emebedding and Tensor Train Decomposition (Shi and Yu, 2018;Chen et al, 2018;Khrulkov et al, 2019;Shu and Nakayama, 2018), require multiple hyper-parameters to be fine-tuned to optimize performance on each dataset. These hyper-parameters influence the number of parameters in the model, and thus the compression rate.…”

“…Additionally, Chen et al (2018) requires an additional optimization step for grouping words, and lacks end-to-end training through back-propagation. Shi and Yu (2018) also requires an additional step for performing k-means clustering for generating the quantization matrix. Thus, most of the current state-of-theart systems are much more complicated to fine-tune for different NLP problems and data-sets.…”

“…Lastly, embedding compression models not based on linear SVD (Khrulkov et al, 2019;Shi and Yu, 2018) require the reconstruction of the entire embedding matrix or additional computations, when used at the output-layer. Thus during runtime, the model either uses the same amount of memory as the uncompressed model or pays a higher computation cost.…”