Csiszár’s Divergences for Non-negative Matrix Factorization: Family of New Algorithms

Abstract

Abstract. In this paper we discus a wide class of loss (cost) functions for non-negative matrix factorization (NMF) and derive several novel algorithms with improved efficiency and robustness to noise and outliers. We review several approaches which allow us to obtain generalized forms of multiplicative NMF algorithms and unify some existing algorithms. We give also the flexible and relaxed form of the NMF algorithms to increase convergence speed and impose some desired constraints such as sparsity and smoothness of components. Moreover, the effects of various regularization terms and constraints are clearly shown. The scope of these results is vast since the proposed generalized divergence functions include quite large number of useful loss functions such as the squared Euclidean distance,Kulback-Leibler divergence, Itakura-Saito, Hellinger, Pearson's chi-square, and Neyman's chi-square distances, etc. We have applied successfully the developed algorithms to blind (or semi blind) source separation (BSS) where sources can be generally statistically dependent, however they satisfy some other conditions or additional constraints such as nonnegativity, sparsity and/or smoothness.

Sparse non-negative tensor factorization using columnwise coordinate descent

“…In addition to the use of different objective functions such as the least squares [4] and Kullback Leibler [13], the main difference among various algorithms lies in the update rule. The update rule directly influences the convergence speed and the quality of the factorization.…”

“…Additionally, the basis can be constrained to be sparse which typically leads to an even more meaningful decomposition of the data. As a result, many researchers focused on sparse non-negative matrix factorization (SNMF) [13,14,4,9] in the past few years.…”

Dependent component analysis for blind restoration of images degraded by turbulent atmosphere

“…Other BSS approaches that can deal with statistically dependent sources include: independent subspace analysis (ISA) [24][25], nonnegative matrix and tensor factorization (NMF/NTF) [27][28][29][30], and the blind Richardson-Lucy (BRL) algorithm [33][34][35][36], which are used for comparison purpose in this paper. They are briefly described as follows.…”

“…NMF/NTF algorithms may yield physically useful solutions by imposing the nonnegativity, sparseness or smoothness constraints on the sources [27][28][29][30]. In [27], the NMF algorithm was first derived to minimize two cost functions: the squared Euclidean distance and the Kullback-Leibler divergence.…”

“…Generalization of the NMF algorithms [27] has been done in [28][29][30]. The gradient-based NMF algorithm with a sparseness constraint being incorporated into the cost function leads to the regularized alternating least square (RALS) algorithm [28]:…”