Initialization Matters

Abstract

Proper initialization is crucial to the optimization and the generalization of neural networks. However, most existing neural recommendation systems initialize the user and item embeddings randomly. In this work, we propose a new initialization scheme for user and item embeddings called Laplacian Eigenmaps with Popularity-based Regularization for Isolated Data (Leporid). Leporid endows the embeddings with information regarding multi-scale neighborhood structures on the data manifold and performs adaptive regularization to compensate for high embedding variance on the tail of the data distribution. Exploiting matrix sparsity, Leporid embeddings can be computed efficiently. We evaluate Leporid in a wide range of neural recommendation models. In contrast to the recent surprising finding that the simple -nearest-neighbor (KNN) method often outperforms neural recommendation systems [8], we show that existing neural systems initialized with Leporid often perform on par or better than KNN. To maximize the effects of the initialization, we propose the Dual-Loss Residual Recommendation (DLR 2 ) network, which, when initialized with Leporid, substantially outperforms both traditional and state-of-the-art neural recommender systems. CCS CONCEPTS• Computing methodologies → Dimensionality reduction and manifold learning; Neural networks; • Information systems → Recommender systems.

A Preprocessing Manifold Learning Strategy Based on t-Distributed Stochastic Neighbor Embedding

“…For decades, a large number of dimensionality reduction methods have been applied to different tasks, among them are Principal Component Analysis (PCA) [10][11][12], Multidimensional Scaling (MDS) [13,14], Sammon Mapping [15], Isomap [16], Locally Linear Embedding (LLE) [17], Laplacian Eigenmaps (LE) [18][19][20], t-Distributed Stochastic Neighbor Embedding (t-SNE) [21][22][23][24] and so on. It is well known that the first three algorithms mentioned above are linear dimensionality reduction methods, which usually break the inner data structure of real-world datasets, thus yielding a poor visualization map.…”