Learning Elastic Embeddings for Customizing On-Device Recommenders

Abstract

In today's context, deploying data-driven services like recommendation on edge devices instead of cloud servers becomes increasingly attractive due to privacy and network latency concerns. A common practice in building compact on-device recommender systems is to compress their embeddings which are normally the cause of excessive parameterization. However, despite the vast variety of devices and their associated memory constraints, existing memory-efficient recommender systems are only specialized for a fixed memory budget in every design and training life cycle, where a new model has to be retrained to obtain the optimal performance while adapting to a smaller/larger memory budget. In this paper, we present a novel lightweight recommendation paradigm that allows a welltrained recommender to be customized for arbitrary device-specific memory constraints without retraining. The core idea is to compose elastic embeddings for each item, where an elastic embedding is the concatenation of a set of embedding blocks that are carefully chosen by an automated search function. Correspondingly, we propose an innovative approach, namely recommendation with universally learned elastic embeddings (RULE). To ensure the expressiveness of all candidate embedding blocks, RULE enforces a diversity-driven regularization when learning different embedding blocks. Then, a performance estimator-based evolutionary search function is designed, allowing for efficient specialization of elastic embeddings under any memory constraint for on-device recommendation. Extensive experiments on real-world datasets reveal the superior performance of RULE under tight memory budgets. CCS CONCEPTS• Information systems → Recommender systems.

AutoML for Deep Recommender Systems: A Survey

“…AutoML for on-device recommender systems automatically designs heterogeneous recommender systems for heterogeneous devices with several restrictions. To deal with one of limitations: memory budget, RULE [107] is proposed to learn the diversified embedding blocks and customise elastic item embeddings for various devices with different memory constraints. AutoML for on-device recommender systems is challenging and distinctive from traditional AutoML or on-device recommender systems, where it emphasises the automated input components and heterology.…”

Thinking inside The Box: Learning Hypercube Representations for Group Recommendation

“…We leverage two metrics, namely recall at rank 𝐾 (Recall@𝐾) [7,10] and normalized discounted cumulative gain at rank 𝐾 (NDCG@𝐾) [9,47] that are widely adopted in recommendation research. We adopt 𝐾 = 10, 20 where all items unvisited by each group are taken as negative samples for evaluation.…”

Single-shot Embedding Dimension Search in Recommender System

“…Recommender systems have been widely deployed to various scenarios such as advertisement [37], online shopping [3], news apps [35], and many others [1,2,21,27,30,32]. The typical inputs of recommender systems are a large number of categorical (e.g., gender) or numerical (e.g., age) features associated with users and items.…”

“…However, most of these methods set a fixed embedding dimension for all features, which could suffer from the following issues: (1) The embeddings could contain tens of billions of parameters resulting in high memory usage and computation cost. (2) Overparameterizing the low-frequency features might induce overfitting and even unexpected noise. On the other hand, high-frequency features need more parameters to convey fruitful information.…”