Graph Construction from Data by Non-Negative Kernel Regression

Abstract

Data driven graph constructions are often used in various applications, including several machine learning tasks, where the goal is to make predictions and discover patterns. However, learning an optimal graph from data is still a challenging task. Weighted K-nearest neighbor and -neighborhood methods are among the most common graph construction methods, due to their computational simplicity but the choice of parameters such as K and associated with these methods is often ad hoc and lacks a clear interpretation. We formulate graph construction as the problem of finding a sparse signal approximation in kernel space, identifying key similarities between methods in signal approximation and existing graph learning methods. We propose non-negative kernel regression (NNK), an improved approach for graph construction with interesting geometric and theoretical properties. We show experimentally the efficiency of NNK graphs, its robustness to choice of sparsity K and better performance over state of the art graph methods in semi supervised learning tasks on real world data.

Representing Deep Neural Networks Latent Space Geometries with Graphs

“…Note that the proposed methodologies use straightforward techniques to build LGGs; thus, they could be enriched with more principled approaches [41,42]. Another area of interest would be to build upon Reference [15] and see what improvements may arise from the use of graph convolutional networks in domains that are not typically supported by graphs.…”

NNK-Means: Data summarization using dictionary learning with non-negative kernel regression

“…We present a data summarization approach using DL that draws ideas from kMeans and our work on NNK neighborhood [11]. We propose a two-stage learning scheme where we iteratively solve sparse coding and dictionary update until convergence, or until a predefined number of iteration or reconstruction error is reached.…”

“…Solving for each Wi in equation ( 4) involves working with a N × N kernel matrix leading to run times that scale poorly with the size of the dataset. However, the geometric understanding of the above nonnegative kernel regression objective in [11], allows us to efficiently solve for the sparse coefficients (Wi) for each data point by selecting and working with a smaller subset of data points. Objective (4) We see that the kSVD approach is unable to adapt to the nonlinear structure of the data and adding a kernel is crucial for the kSVD approach to perform well.…”

“…The starting point for our DL method is our graph construction framework using non-negative kernel regression (NNK) [11,12]. NNK formulates neighborhoods as a signal representation problem, where each data point (represented as a function in RKHS) is to be approximated by functions corresponding to its neighboring points, i.e.,…”

“…It is important to note that while DL has witnessed great progress and success in a range of applications, the use of dictionary learning methods for data summarization has not been considered before. In order to design a dictionary with geometry properties resembling those of kMeans, we leverage our work on neighborhood definition with non-negative kernel regression (NNK) [11,12] and propose a novel NNK-Means dictionary learning technique for data summarization. NNK-Means learns dictionaries optimized for higher sparsity with atoms that belong to the given data manifold as in the kMeans approach.…”