Minimax Lower Bounds on Dictionary Learning for Tensor Data

Shakeri, Zahra; Bajwa, Waheed U.; Sarwate, Anand D.

doi:10.1109/tit.2018.2799931

Cited by 20 publications

(17 citation statements)

References 38 publications

(97 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…a lower bound for the minimax risk ε * is attained. A formal proof of Theorem 1 relies on the following lemmas whose proofs appear in the full version of this work [17]. Note that since our construction of D L is more complex than the vector case [16, Theorem 1], it requires a different sequence of lemmas, with the exception of Lemma 3, which follows from the vector case.…”

Section: Lower Bound For General Coefficientsmentioning

confidence: 99%

See 1 more Smart Citation

Minimax lower bounds for Kronecker-structured dictionary learning

Shakeri¹,

Bajwa²,

Sarwate³

2016

2016 IEEE International Symposium on Information Theory (ISIT)

Self Cite

View full text Add to dashboard Cite

Abstract-Dictionary learning is the problem of estimating the collection of atomic elements that provide a sparse representation of measured/collected signals or data. This paper finds fundamental limits on the sample complexity of estimating dictionaries for tensor data by proving a lower bound on the minimax risk. This lower bound depends on the dimensions of the tensor and parameters of the generative model. The focus of this paper is on second-order tensor data, with the underlying dictionaries constructed by taking the Kronecker product of two smaller dictionaries and the observed data generated by sparse linear combinations of dictionary atoms observed through white Gaussian noise. In this regard, the paper provides a general lower bound on the minimax risk and also adapts the proof techniques for equivalent results using sparse and Gaussian coefficient models. The reported results suggest that the sample complexity of dictionary learning for tensor data can be significantly lower than that for unstructured data.

show abstract

Section: Lower Bound For General Coefficientsmentioning

confidence: 99%

“…We state a variation of Lemma 2 necessary for the proof of Theorem 2. The proof of the lemma is again provided in [17].…”

Section: A Sparse Gaussian Coefficientsmentioning

confidence: 99%

Minimax lower bounds for Kronecker-structured dictionary learning

Shakeri¹,

Bajwa²,

Sarwate³

2016

2016 IEEE International Symposium on Information Theory (ISIT)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Theoretical analysis suggests that the sample complexity of tensor-structured dictionary learning can be significantly lower than that for unstructured, vector data; see [113] for 2D data, and [114] for N -order tensor data. This suggests better performance is achievable with separable dictionary learning from tensor data compared to vector-based dictionary learning methods.…”

Section: E Tensor-structured Dictionary Learningmentioning

confidence: 99%

Tensor Methods in Computer Vision and Deep Learning

et al. 2021

View full text Add to dashboard Cite

Tensors, or multidimensional arrays, are data structures that can naturally represent visual data of multiple dimensions. Inherently able to efficiently capture structured, latent semantic spaces and high-order interactions, tensors have a long history of applications in a wide span of computer vision problems. With the advent of the deep learning paradigm shift in computer vision, tensors have become even more fundamental. Indeed, essential ingredients in modern deep learning architectures, such as convolutions and attention mechanisms, can readily be considered as tensor mappings. In effect, tensor methods are increasingly finding significant applications in deep learning, including the design of memory and compute efficient network architectures, improving robustness to random noise and adversarial attacks, and aiding the theoretical understanding of deep networks.This article provides an in-depth and practical review of tensors and tensor methods in the context of representation learning and deep learning, with a particular focus on visual data analysis and computer vision applications. Concretely, besides fundamental work in tensor-based visual data analysis methods, we focus on recent developments that have brought on a gradual increase of tensor methods, especially in deep learning architectures, and their implications in computer vision applications. To further enable the newcomer to grasp such concepts quickly, we provide companion Python notebooks, covering key aspects of the paper and implementing them, step-by-step with TensorLy.

show abstract

“…Beyond scalability, separable dictionaries have been shown to be theoretically appealing when dealing with tensor-valued observations. According to [24], the necessary number of samples for accurate (up to a given error) reconstruction of a Kronecker-structured dictionary within a local neighborhood scales with the sum of the product of the dimensions of the constituting dictionaries when learning on tensor data (see [25] for 2-dimensional data, and [26], [27] for N-order tensor data), compared to the product for vectorized observations. This suggests better performance is achievable compared to classical methods on tensor observations.…”

Section: Limits Of the Classical Approachmentioning

confidence: 99%

Robust Kronecker Component Analysis

Bahri

Panagakis

Zafeiriou

2019

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Dictionary learning and component analysis models are fundamental for learning compact representations that are relevant to a given task (feature extraction, dimensionality reduction, denoising, etc.). The model complexity is encoded by means of specific structure, such as sparsity, low-rankness, or nonnegativity. Unfortunately, approaches like K-SVD-that learn dictionaries for sparse coding via Singular Value Decomposition (SVD)-are hard to scale to high-volume and high-dimensional visual data, and fragile in the presence of outliers. Conversely, robust component analysis methods such as the Robust Principal Component Analysis (RPCA) are able to recover low-complexity (e.g., low-rank) representations from data corrupted with noise of unknown magnitude and support, but do not provide a dictionary that respects the structure of the data (e.g., images), and also involve expensive computations. In this paper, we propose a novel Kronecker-decomposable component analysis model, coined as Robust Kronecker Component Analysis (RKCA), that combines ideas from sparse dictionary learning and robust component analysis. RKCA has several appealing properties, including robustness to gross corruption; it can be used for low-rank modeling, and leverages separability to solve significantly smaller problems. We design an efficient learning algorithm by drawing links with a restricted form of tensor factorization, and analyze its optimality and low-rankness properties. The effectiveness of the proposed approach is demonstrated on real-world applications, namely background subtraction and image denoising and completion, by performing a thorough comparison with the current state of the art.

show abstract

Minimax Lower Bounds on Dictionary Learning for Tensor Data

Cited by 20 publications

References 38 publications

Minimax lower bounds for Kronecker-structured dictionary learning

Minimax lower bounds for Kronecker-structured dictionary learning

Tensor Methods in Computer Vision and Deep Learning

Robust Kronecker Component Analysis

Contact Info

Product

Resources

About