Scaling Law for Recommendation Models: Towards General-purpose User Representations

Shin, Kyuyong; Kwak, Hanock; Kim, Kyung-Min; Kim, Su Young; Ramstrom, Max Nihlen

doi:10.48550/arxiv.2111.11294

Cited by 2 publications

(2 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, these models are all based on the shared-ID assumption in source and target domains, which tends to be difficult to hold in practice. Distinct from these work, some preprint paper [34,35,41] devised general-purpose gpRS by leveraging the textual information. [39] learned user representations based on text and image modalities with images processed into frozen features beforehand.…”

Section: Related Workmentioning

confidence: 99%

TransRec: Learning Transferable Recommendation from Mixture-of-Modality Feedback

Wang¹,

Yuan²,

Cheng³

et al. 2022

Preprint

View full text Add to dashboard Cite

Learning big models and then transfer has become the de facto practice in computer vision (CV) and natural language processing (NLP). However, such unified paradigm is uncommon for recommender systems (RS). A critical issue that hampers this is that standard recommendation models are built on unshareable identity data, where both users and their interacted items are represented by unique IDs. In this paper, we study a novel scenario where user's interaction feedback involves mixture-of-modality (MoM) items. We present TransRec, a straightforward modification done on the popular ID-based RS framework. TransRec directly learns from MoM feedback in an end-to-end manner, and thus enables effective transfer learning under various scenarios without relying on overlapped users or items. We empirically study the transferring ability of TransRec across four different real-world recommendation settings. Besides, we study its effects by scaling the size of source and target data. Our results suggest that learning recommenders from MoM feedback provides a promising way to realize universal recommender systems. Our code and datasets will be made available. * Work was done when Jie Wang was a visiting scholar at Westlake University and intern at Platform and Content Group, Tencent.† Corresponding author. Fajie Yuan designed the research and Jie Wang led the experiments.Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

TransRec: Learning Transferable Recommendation from Mixture-of-Modality Feedback

Wang¹,

Yuan²,

Cheng³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Training deep networks is a canonical task in modern machine learning. Training and serving deep networks with very large parameter counts efficiently is of paramount importance in recent years, since significant performance gains on a variety of tasks are possible simply by scaling up parameter counts [Rae et al, 2021, Shin et al, 2021, Gordon et al, 2021, Sharma and Kaplan, 2020. Further theoretical evidence suggests that requiring the resulting learned networks to be smooth functions (and therefore, in some sense, robust to perturbations) entails even larger parameter counts [Bubeck and Sellke, 2021].…”

Section: Introductionmentioning

confidence: 99%

Nonlinear Initialization Methods for Low-Rank Neural Networks

Vodrahalli¹,

Shivanna²,

Sathiamoorthy³

et al. 2022

Preprint

View full text Add to dashboard Cite

We study algorithms for learning low-rank neural networks -networks where the weight parameters are re-parameterized by products of two low-rank matrices. First, we present a provably efficient algorithm which learns an optimal low-rank approximation to a single-hidden-layer ReLU network up to additive error with probability ≥ 1 − δ, given access to noiseless samples with Gaussian marginals in polynomial time and samples. Thus, we provide the first example of an algorithm which can efficiently learn a neural network up to additive error with respect to a strictly smaller hypothesis class. To solve this problem, we introduce an efficient SVD-based Nonlinear Kernel Projection algorithm for solving a nonlinear low-rank approximation problem over Gaussian space. Inspired by the efficiency of our algorithm, we propose a novel low-rank initialization framework for training low-rank deep networks, and prove that for ReLU networks, the gap between our method and existing schemes widens as the desired rank of the approximating weights decreases, or as the dimension of the inputs increases (the latter point holds when network width is superlinear in dimension). Finally, we validate our theory by training ResNet and EfficientNet models [

show abstract

Scaling Law for Recommendation Models: Towards General-purpose User Representations

Cited by 2 publications

References 20 publications

TransRec: Learning Transferable Recommendation from Mixture-of-Modality Feedback

TransRec: Learning Transferable Recommendation from Mixture-of-Modality Feedback

Nonlinear Initialization Methods for Low-Rank Neural Networks

Contact Info

Product

Resources

About