Robust Decentralized Low-Rank Matrix Decomposition

Abstract

Low-rank matrix approximation is an important tool in data mining with a wide range of applications including recommender systems, clustering, and identifying topics in documents. When the matrix to be approximated originates from a large distributed system-such as a network of mobile phones or smart meters-this is a very challenging problem due to the strongly conflicting, yet essential requirements of efficiency, robustness, and privacy preservation. We argue that while collecting sensitive data in a centralized fashion may be efficient, it is not an option when considering privacy and efficiency at the same time. Thus, we do not allow any sensitive data to leave the nodes of the network. The local information at each node (personal attributes, documents, media ratings, etc.) defines one row in the matrix. This means that all computations have to be performed at the edge of the network. Known parallel methods that respect the locality constraint-such as synchronized parallel gradient search or distributed iterative methods-require synchronized rounds or have inherent issues with load balancing, thus they are not robust to failure. Our distributed stochastic gradient descent algorithm overcomes these limitations. During the execution, any sensitive information remains local, while the global features (e.g., the factor model of movies) converge to the correct value at all nodes. We present a theoretical derivation, as well as a thorough experimental evaluation of our algorithm. We demonstrate that the convergence speed of our method is competitive while not relying on synchronization and being robust to extreme and realistic failure scenarios. To demonstrate the feasibility of our approach, we present trace-based simulations, real smartphone user behavior analysis, and tests over real movie recommender system data.

Decentralized Recommendation Based on Matrix Factorization: A Comparison of Gossip and Federated Learning

“…Our targeted application environment consists of a potentially large set of personal devices holding private data. We follow the approach in our previous paper [9] and we will adapt the same approach to federated learning. We shall assume that each row in matrix A is stored on exactly one device.…”

“…For this reason, the local update calculation is more thorough, and compression techniques can be applied when uploading the updates to the server. Gossip learning has also been proposed to address the same challenge [9,14]. This approach is fully decentralized, no parameter server is necessary.…”

“…To do this, we implement a recommender system in both paradigms, based on low-rank matrix decomposition. The gossip learning implementation is based on our previous work [9]. We propose a federated learning implementation as well, following the same design, but adapted to the centralized communication pattern.…”

“…To sum up our key original contributions in the present study: (1) we propose an efficient collaborative filtering method for federated learning; (2) we improve several details of our previous solution [9] as well including the introduction of coordinate-based age parameters to manage aggregation and the application of an optimized version of subsampling to gossip learning; and (3) we compare the two methods empirically based on a realistic churn trace collected by the application Stunner [2].…”

Decentralized Management of Random Walks over a Mobile Phone Network

“…Here, the model that is being fit on the data performs a uniform random walk over the network and it is updated before each step using the local data. Recently, the same idea has been applied to matrix factorization as well that is useful, for example, in implementing decentralized recommender systems [2].…”