Stochastic Shared Embeddings: Data-driven Regularization of Embedding Layers

Abstract

In deep neural nets, lower level embedding layers account for a large portion of the total number of parameters. Tikhonov regularization, graph-based regularization, and hard parameter sharing are approaches that introduce explicit biases into training in a hope to reduce statistical complexity. Alternatively, we propose stochastically shared embeddings (SSE), a data-driven approach to regularizing embedding layers, which stochastically transitions between embeddings during stochastic gradient descent (SGD). Because SSE integrates seamlessly with existing SGD algorithms, it can be used with only minor modifications when training large scale neural networks. We develop two versions of SSE: SSE-Graph using knowledge graphs of embeddings; SSE-SE using no prior information. We provide theoretical guarantees for our method and show its empirical effectiveness on 6 distinct tasks, from simple neural networks with one hidden layer in recommender systems, to the transformer and BERT in natural languages. We find that when used along with widely-used regularization methods such as weight decay and dropout, our proposed SSE can further reduce overfitting, which often leads to more favorable generalization results.Preprint. Under review.

Temporal Collaborative Ranking Via Personalized Transformer

“…[16] found that adding additional personalized embeddings did not improve the performance of their Transformer model, and postulate that this is due to the fact that they already use the user history and the embeddings only contribute to overfitting. Although introducing user embeddings into the model is indeed difficult with existing regularization techniques for embeddings, we show that personalization can greatly improve ranking performances with recent regularization technique called Stochastic Shared Embeddings (SSE) [41]. The personalized Transformer (SSE-PT) model with SSE regularization works well for all 5 realworld datasets we consider, outperforming previous state-of-the-art algorithm SASRec by almost 5% in terms of NDCG@10.…”

“…There are many other regularization techniques, including parameter sharing [5], max-norm regularization [31], gradient clipping [25], etc. Very recently, a new regularization technique called Stochastic Shared Embeddings (SSE) [41] is proposed as a new means of regularizing embedding layers. [41] develops two versions of SSE, SSE-Graph and SSE-SE.…”

“…Very recently, a new regularization technique called Stochastic Shared Embeddings (SSE) [41] is proposed as a new means of regularizing embedding layers. [41] develops two versions of SSE, SSE-Graph and SSE-SE. We find that SSE-SE is essential to the success of our Personalized Transformer (SSE-PT) model.…”

“…Unlike previous SASRec model [16], we use one more regularization technique in our SSE-PT model specifically for embedding layer in addition to the ones listed earlier: the Stochastic Shared Embeddings (SSE) [41]. The reason that we want to use this additional regularization technique is that we find the existing well-known regularization techniques like layer normalization, dropout and weight decay cannot prevent the model from over-fitting badly after introducing user embeddings.…”

“…The main idea of SSE is to stochastically replace embeddings with another embedding during SGD, which has the effect of regularizing the embedding layers. Specifically, SSE-SE replaces one embedding with another embedding stochastically with probability p, which is called SSE probability in [41]. There are 3 different places in our model that SSE-SE can be applied.…”