Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems

Ginart, Antonio; Naumov, Maxim; Mudigere, Dheevatsa; Yang, Jiyan; Zou, James

doi:10.48550/arxiv.1909.11810

Cited by 20 publications

(32 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Embedding Usage. When embedding vectors are assigned with different dimensions, all of the existing methods [4,8,12,18,19] have to design an additional layer to unify these vectors to a same length to fit the following uniform MLP layers in DLRMs. Unlike these methods, our method does not need any additional layers since the masked embedding vectors ê𝑖 have the same length by zeros paddings, and can be directly fed into the following layers.…”

Section: Methods 21 Basic Ideamentioning

confidence: 99%

“…They can be primarily divided into two categories. (1) Rule-based methods adopt human-defined rules, typically according to the feature frequencies, to give different embedding dimensions to different feature values[4] (see Fig.1(b) for an example). The problem with this category of methods is that they heavily rely on human knowledge and human labor.…”

mentioning

confidence: 99%

“…We consider different kinds of state-of-the-art embedding methods as baselines (1) Standard: traditional Fixed-based Embedding (FBE). (2) Rule-Based: MDE[4] divides different feature values into several blocks by their frequency, and assigns different embedding dimensions for different blocks by rules. (3) NAS-Based: AutoEmb[19] adopts NAS to select embedding dimensions among some candidate embedding dimensions for different feature values.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Learning Effective and Efficient Embedding via an Adaptively-Masked Twins-based Layer

Yan

Wang

Zhang

et al. 2021

Proceedings of the 30th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

Embedding learning for categorical features is crucial for the deep learning-based recommendation models (DLRMs). Each feature value is mapped to an embedding vector via an embedding learning process. Conventional methods configure a fixed and uniform embedding size to all feature values from the same feature field. However, such a configuration is not only sub-optimal for embedding learning but also memory costly. Existing methods that attempt to resolve these problems, either rule-based or neural architecture search (NAS)-based, need extensive efforts on the human design or network training. They are also not flexible in embedding size selection or in warm-start-based applications. In this paper, we propose a novel and effective embedding size selection scheme. Specifically, we design an Adaptively-Masked Twins-based Layer (AMTL) behind the standard embedding layer. AMTL generates a mask vector to mask the undesired dimensions for each embedding vector. The mask vector brings flexibility in selecting the dimensions and the proposed layer can be easily added to either untrained or trained DLRMs. Extensive experimental evaluations show that the proposed scheme outperforms competitive baselines on all the benchmark tasks, and is also memory-efficient, saving 60% memory usage without compromising any performance metrics. CCS CONCEPTS• Information systems → Recommender systems.

show abstract

Section: Methods 21 Basic Ideamentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Learning Effective and Efficient Embedding via an Adaptively-Masked Twins-based Layer

Yan

Wang

Zhang

et al. 2021

Proceedings of the 30th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

show abstract

“…In most approaches, the embedding module allocates variable-length embeddings or multi embedding to different features. Such methods includes NIS [12], Mixed-dimension [6], and AutoEmb [27]. More models pay attention to model interaction between features in an implicit or an explicit manner based on the basic embedding algorithms.…”

Section: Related Workmentioning

confidence: 99%

FINT: Field-aware INTeraction Neural Network For CTR Prediction

Zhao¹,

Yang²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

As a critical component for online advertising and marking, clickthrough rate (CTR) prediction has draw lots of attentions from both industry and academia field. Recently, the deep learning has become the mainstream methodological choice for CTR. Despite of sustainable efforts have been made, existing approaches still pose several challenges. On the one hand, high-order interaction between the features is under-explored. On the other hand, high-order interactions may neglect the semantic information from the low-order fields. In this paper, we proposed a novel prediction method, named FINT, that employs the Field-aware INTeraction layer which captures high-order feature interactions while retaining the low-order field information. To empirically investigate the effectiveness and robustness of the FINT, we perform extensive experiments on the three realistic databases: KDD2012, Criteo and Avazu. The obtained results demonstrate that the FINT can significantly improve the performance compared to the existing methods, without increasing the amount of computation required. Moreover, the proposed method brought about 2.72% increase to the advertising revenue of a big online video app through A/B testing. To better promote the research in CTR field, we released our code as well as reference implementation at: https://github.com/zhishan01/FINT. CCS CONCEPTS• Information systems → Retrieval models and ranking; Information retrieval.

show abstract

“…In general, there are two directions to reduce the embedding table size, i.e., reducing the size of each embedding vector and reducing the number (i.e., 𝑁 ) of the embedding vectors in an embedding table. The embedding table size of the former methods (e.g., product quantization [5,9], K-D method [2,11,13,15,20], and AutoDim [6,10,16,17,28,29]) is still linearly increased with |𝐹 |, failing to tackle the memory problem caused by a large vocabulary size in web-scale applications [19]. Hence these methods are not considered in our paper.…”

Section: Introductionmentioning

confidence: 99%

Binary Code based Hash Embedding for Web-scale Applications

Yan

Wang

Liu

et al. 2021

Proceedings of the 30th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

Nowadays, deep learning models are widely adopted in web-scale applications such as recommender systems, and online advertising. In these applications, embedding learning of categorical features is crucial to the success of deep learning models. In these models, a standard method is that each categorical feature value is assigned a unique embedding vector which can be learned and optimized. Although this method can well capture the characteristics of the categorical features and promise good performance, it can incur a huge memory cost to store the embedding table, especially for those web-scale applications. Such a huge memory cost significantly holds back the effectiveness and usability of EDRMs. In this paper, we propose a binary code based hash embedding method which allows the size of the embedding table to be reduced in arbitrary scale without compromising too much performance. Experimental evaluation results show that one can still achieve 99% performance even if the embedding table size is reduced 1000× smaller than the original one with our proposed method. CCS CONCEPTS• Information systems → Recommender systems.

show abstract

Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems

Cited by 20 publications

References 44 publications

Learning Effective and Efficient Embedding via an Adaptively-Masked Twins-based Layer

Learning Effective and Efficient Embedding via an Adaptively-Masked Twins-based Layer

FINT: Field-aware INTeraction Neural Network For CTR Prediction

Binary Code based Hash Embedding for Web-scale Applications

Contact Info

Product

Resources

About