eTREE: Learning Tree-structured Embeddings

Abstract

Matrix factorization (MF) plays an important role in a wide range of machine learning and data mining models. MF is commonly used to obtain item embeddings and feature representations due to its ability to capture correlations and higher-order statistical dependencies across dimensions. In many applications, the categories of items exhibit a hierarchical tree structure. For instance, human diseases can be divided into coarse categories, e.g., bacterial, and viral. These categories can be further divided into finer categories, e.g., viral infections can be respiratory, gastrointestinal, and exanthematous viral diseases. In e-commerce, products, movies, books, etc., are grouped into hierarchical categories, e.g., clothing items are divided by gender, then by type (formal, casual, etc.). While the tree structure and the categories of the different items may be known in some applications, they have to be learned together with the embeddings in many others. In this work, we propose eTREE, a model that incorporates the (usually ignored) tree structure to enhance the quality of the embeddings. We leverage the special uniqueness properties of Nonnegative MF (NMF) to prove identifiability of eTREE. The proposed model not only exploits the tree structure prior, but also learns the hierarchical clustering in an unsupervised data-driven fashion. We derive an efficient algorithmic solution and a scalable implementation of eTREE that exploits parallel computing, computation caching, and warm start strategies. We showcase the effectiveness of eTREE on real data from various application domains: healthcare, recommender systems, and education. We also demonstrate the meaningfulness of the tree obtained from eTREE by means of domain experts interpretation.

Clustered Embedding Learning for Recommender Systems

“…Enforcing such a condition in our implementation has improved the model performance. The assumption of B 𝑞 being sufficiently scattered is common and is likely to be satisfied as shown by [1]. Based on Theorem 1, we can further deduce the existence of a unique optimal number of clusters: PROPOSITION 1.…”

“…One line of research regularizes the embeddings to follow a given clustering structure to improve their qualities. For example, Joint NMF and K-Means (JNKM) [25] forces the embeddings to follow a K-clustering structure; eTree [1] learns the embeddings under an implicit tree (i.e., hierarchical clustering). Prior works in this line also include HSR [21], HieVH [18], etc.…”

“…The formal definitions (e.g., sufficiently scattered) [1,5,25] and detailed proofs are provided in Appendix B.…”

“…To mitigate the problem of cold users and items, a common idea is to impose regularization, e.g., forcing all embeddings to follow a given clustering structure [1,25]. However, using a fixed predefined clustering structure lacks flexibility; the given clustering structure can hardly be optimal.…”

“…However, using a fixed predefined clustering structure lacks flexibility; the given clustering structure can hardly be optimal. Besides, existing methods [1,25] require training in full embedding, still suffering from high memory costs.…”