Scalable Nonnegative Matrix Factorization with Block-wise Updates

Yin, Jiangtao; Gao, Lixin; Zhang, Zhongfei Mark

doi:10.1007/978-3-662-44845-8_22

Cited by 24 publications

(21 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The general techniques of incremental updates have shown efficiency in many algorithms, such as Nonnegative Matrix Factorization [27] and Expectation-Maximization [28]. In this section, we present an incremental update mechanism for BP algorithms, referred to as an incremental-update approach.…”

Section: Incremental Updatesmentioning

confidence: 99%

Scalable Distributed Belief Propagation with Prioritized Block Updates

Yin

Gao

2014

Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management

Self Cite

View full text Add to dashboard Cite

Belief propagation (BP) is a popular method for performing approximate inference on probabilistic graphical models. However, its message updates are time-consuming, and the schedule for updating messages is crucial to its running time and even convergence. In this paper, we propose a new scheduling scheme that selects a set of messages to update at a time and leverages a novel priority to determine which messages are selected. Additionally, an incremental update approach is introduced to accelerate the computation of the priority. As the size of the model grows, it is desirable to leverage the parallelism of a cluster of machines to reduce the inference time. Therefore, we design a distributed framework, Prom, to facilitate the implementation of BP algorithms. We evaluate the proposed scheduling scheme (supported by Prom) via extensive experiments on a local cluster as well as the Amazon EC2 cloud. The evaluation results show that our scheduling scheme outperforms the state-of-the-art counterpart.

show abstract

Section: Incremental Updatesmentioning

confidence: 99%

Scalable Distributed Belief Propagation with Prioritized Block Updates

Yin

Gao

2014

Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management

Self Cite

View full text Add to dashboard Cite

show abstract

“…Assuming that the objective function Ψ( Y || A X ) is expressed by an additive function, eg, the Bregman divergence, then we have:

normalΨ false(bold-italicY false‖ bold-italicA bold-italicX false) = \sum_{m = 1}^{M} \sum_{n = 1}^{N} normalΨ false(Y_{m n} false‖ A_{m} X_{n} false)

. Several optimization strategies aim to perform blockwise updates via the sequential minimization of Ψ( Y || A X ) with respect to the corresponding blocks A m or X n . In the next subsection, we propose a different approach to blockwise updates based on three‐step minimization.…”

Section: Distributed Nmfmentioning

confidence: 99%

Distributed geometric nonnegative matrix factorization and hierarchical alternating least squares–based nonnegative tensor factorization with the MapReduce paradigm

Zdunek

Fonał

2018

Concurrency and Computation

View full text Add to dashboard Cite

Summary Nonnegative matrix factorization and its multilinear extension known as nonnegative tensor factorization are commonly used methods in machine learning and data analysis for feature extraction and dimensionality reduction for nonnegative high‐dimensional data. Dimensionality reduction for massive amounts of data usually involves distributed computation across multi‐node computer architectures. In this study, we propose various computational strategies for parallel and distributed computation of the latent factors in both factorization models, all of which are based on partitioning the computational tasks according to the MapReduce paradigm. We extend the previously reported distributed hierarchical alternating least squares algorithm to the multi‐way array factorization model, where we assume that the observed multi‐way data can be partitioned into chunks along one mode. Moreover, we propose a new geometry‐based distributed computational strategy for solving nonnegative matrix factorization problems. Numerical experiments performed using various large‐scale data sets demonstrated that these algorithms are efficient and robust to noisy data.

show abstract

“…There are several recent distributed NMF algorithms in the literature [19,6,32,20]. Liu et al propose running Multiplicative Update (MU) for KL divergence, squared loss, and "exponential" loss functions [20].…”

Section: Surveymentioning

confidence: 99%

“…Using similar approaches, Liao et al implement an open source Hadoop-based MU algorithm and study its scalability on large-scale biological data sets [19]. Also, Yin, Gao, and Zhang present a scalable NMF that can perform frequent updates, which aim to use the most recently updated data [32]. Similarly Faloutsos et al propose a distributed, scalable method for decomposing matrices, tensors, and coupled data sets through stochastic gradient descent on a variety of objective functions [6].…”

Section: Surveymentioning

confidence: 99%

Partitioning and Communication Strategies for Sparse Non-negative Matrix Factorization

Kaya

Kannan

Ballard

2018

Proceedings of the 47th International Conference on Parallel Processing

View full text Add to dashboard Cite

Non-negative matrix factorization (NMF), the problem of finding two non-negative low-rank factors whose product approximates an input matrix, is a useful tool for many data mining and scientific applications such as topic modeling in text mining and blind source separation in microscopy. In this paper, we focus on scaling algorithms for NMF to very large sparse datasets and massively parallel machines by employing effective algorithms, communication patterns, and partitioning schemes that leverage the sparsity of the input matrix. In the case of machine learning workflow, the computations after SpMM must deal with dense matrices, as Sparse-Dense matrix multiplication will result in a dense matrix. Hence, the partitioning strategy considering only SpMM will result in a huge imbalance in the overall workflow especially on computations after SpMM and in this specific case of NMF on non-negative least squares computations. Towards this, we consider two previous works developed for related problems, one that uses a fine-grained partitioning strategy using a point-to-point communication pattern and on that uses a checkerboard partitioning strategy using a collective-based communication pattern. We show that a combination of the previous approaches balances the demands of the various computations within NMF algorithms and achieves high efficiency and scalability. From the experiments, we could see that our proposed algorithm communicates atleast 4x less than the collective and achieves upto 100x speed up over the baseline FAUN on real world datasets. Our algorithm was experimented in two different super computing platforms and we could scale up to 32000 processors on Bluegene/Q. Key-words: sparse non-negative matrix factorization, hypergraph partitioning, parallel algorithms * For the published version of this research report, please refer to https://doi.

show abstract

Scalable Nonnegative Matrix Factorization with Block-wise Updates

Cited by 24 publications

References 23 publications

Scalable Distributed Belief Propagation with Prioritized Block Updates

Scalable Distributed Belief Propagation with Prioritized Block Updates

Distributed geometric nonnegative matrix factorization and hierarchical alternating least squares–based nonnegative tensor factorization with the MapReduce paradigm

Partitioning and Communication Strategies for Sparse Non-negative Matrix Factorization

Contact Info

Product

Resources

About