Finding the Sparsest Vectors in a Subspace: Theory, Algorithms, and Applications

Qu, Qing; Zhu, Zhihui; Li, Xiao; Tsakiris, Manolis C.; Wright, John; Vidal, René

doi:10.48550/arxiv.2001.06970

Cited by 5 publications

(9 citation statements)

References 115 publications

(262 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, from a boarder perspective our work is rooted in recent advances on global nonconvex optimization theory for signal processing and machine learning problems [52,[78][79][80][81]. In a sequence of works [78,80,[82][83][84][85][86][87][88][89][90][91][92][93][94][95], the authors showed that many problems exhibit "equivalently good" global minimizers due to symmetries and intrinsic low-dimensional structures, and the loss functions are usually strict saddles [50][51][52]. These problems include, but are not limited to, phase retrieval [86,87], low-rank matrix recovery [78,82,85,88,90], dictionary learning [80,83,84,91,96], and sparse blind deconvolution [92][93][94][95].…”

Section: Bias Term Results Constraint Weight Decaymentioning

confidence: 99%

“…In a sequence of works [78,80,[82][83][84][85][86][87][88][89][90][91][92][93][94][95], the authors showed that many problems exhibit "equivalently good" global minimizers due to symmetries and intrinsic low-dimensional structures, and the loss functions are usually strict saddles [50][51][52]. These problems include, but are not limited to, phase retrieval [86,87], low-rank matrix recovery [78,82,85,88,90], dictionary learning [80,83,84,91,96], and sparse blind deconvolution [92][93][94][95]. As we shall see, the global minimizers (i.e., simplex ETFs) of our problem here also exhibit a similar rotational symmetry, compared to low-rank matrix recovery.…”

Section: Bias Term Results Constraint Weight Decaymentioning

confidence: 99%

“…• Relationship to Low-Rank Matrix Recovery. As discussed in Section 1, it has been recently shown that the strict saddle property holds for a wide range of nonconvex problems in machine learning [78,80,[82][83][84][85][86][87][88][89][90][91][92]96], including low-rank matrix recovery [78,89,[100][101][102][103]. As we know that [35] for a proof), our formulation in ( 4) is closely related to low-rank matrix problems [78,89,[100][101][102][103] with the Burer-Moneirto factorization approach [104], by viewing W and H as two factors of a matrix Z = W H. The differences lie in the loss functions and statistical properties of the problem.…”

Section: Remarkmentioning

confidence: 99%

See 2 more Smart Citations

A Geometric Analysis of Neural Collapse with Unconstrained Features

Zhu,

Ding,

Zhou

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We provide the first global optimization landscape analysis of Neural Collapse -an intriguing empirical phenomenon that arises in the last-layer classifiers and features of neural networks during the terminal phase of training. As recently reported in [1], this phenomenon implies that (i) the class means and the last-layer classifiers all collapse to the vertices of a Simplex Equiangular Tight Frame (ETF) up to scaling, and (ii) cross-example within-class variability of last-layer activations collapses to zero. We study the problem based on a simplified unconstrained feature model, which isolates the topmost layers from the classifier of the neural network. In this context, we show that the classical cross-entropy loss with weight decay has a benign global landscape, in the sense that the only global minimizers are the Simplex ETFs while all other critical points are strict saddles whose Hessian exhibit negative curvature directions. In contrast to existing landscape analysis for deep neural networks which is often disconnected from practice, our analysis of the simplified model not only does it explain what kind of features are learned in the last layer, but it also shows why they can be efficiently optimized in the simplified settings, matching the empirical observations in practical deep network architectures. These findings could have profound implications for optimization, generalization, and robustness of broad interests. For example, our experiments demonstrate that one may set the feature dimension equal to the number of classes and fix the last-layer classifier to be a Simplex ETF for network training, which reduces memory cost by over 20% on ResNet18 without sacrificing the generalization performance.

show abstract

Section: Bias Term Results Constraint Weight Decaymentioning

confidence: 99%

Section: Bias Term Results Constraint Weight Decaymentioning

confidence: 99%

Section: Remarkmentioning

confidence: 99%

See 1 more Smart Citation

A Geometric Analysis of Neural Collapse with Unconstrained Features

Zhu,

Ding,

Zhou

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…To deal with this challenge, in the following we further investigate the global optimization landscape of (3). By leveraging recent advances on nonconvex optimization [32][33][34][39][40][41][42], we first show that our nonconvex MSE loss (3) without bias term is a strict saddle function that every non-global critical point is a saddle point with negative curvature (i.e., its Hessian has at least one negative eigenvalue).…”

Section: Characterizations Of the Benign Global Landscapementioning

confidence: 99%

On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features

Zhou¹,

Li²,

Ding³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

When training deep neural networks for classification tasks, an intriguing empirical phenomenon has been widely observed in the last-layer classifiers and features, where (i) the class means and the last-layer classifiers all collapse to the vertices of a Simplex Equiangular Tight Frame (ETF) up to scaling, and (ii) cross-example within-class variability of last-layer activations collapses to zero. This phenomenon is called Neural Collapse (NC), which seems to take place regardless of the choice of loss functions. In this work, we justify NC under the mean squared error (MSE) loss, where recent empirical evidence shows that it performs comparably or even better than the de-facto cross-entropy loss. Under a simplified unconstrained feature model, we provide the first global landscape analysis for vanilla nonconvex MSE loss and show that the (only!) global minimizers are neural collapse solutions, while all other critical points are strict saddles whose Hessian exhibit negative curvature directions. Furthermore, we justify the usage of rescaled MSE loss by probing the optimization landscape around the NC solutions, showing that the landscape can be improved by tuning the rescaling hyperparameters. Finally, our theoretical findings are experimentally verified on practical network architectures.

show abstract

“…The form in ( 4) is closely related to recent work on blind deconvolution [45,50]. More specifically, the work showed that normalizing the output z out via preconditioning eliminates bad local minimizers and dramatically improves the nonconvex optimization landscapes for learning the kernel a.…”

Section: A Warm-up Studymentioning

confidence: 95%

Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training

Liu¹,

Li²,

Zhai³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Normalization techniques have become a basic component in modern convolutional neural networks (ConvNets). In particular, many recent works demonstrate that promoting the orthogonality of the weights helps train deep models and improve robustness. For ConvNets, most existing methods are based on penalizing or normalizing weight matrices derived from concatenating or flattening the convolutional kernels. These methods often destroy or ignore the benign convolutional structure of the kernels; therefore, they are often expensive or impractical for deep ConvNets. In contrast, we introduce a simple and efficient "convolutional normalization" method that can fully exploit the convolutional structure in the Fourier domain and serve as a simple plug-and-play module to be conveniently incorporated into any ConvNets. Our method is inspired by recent work on preconditioning methods for convolutional sparse coding and can effectively promote each layer's channel-wise isometry. Furthermore, we show that convolutional normalization can reduce the layerwise spectral norm of the weight matrices and hence improve the Lipschitzness of the network, leading to easier training and improved robustness for deep ConvNets. Applied to classification under noise corruptions and generative adversarial network (GAN), we show that convolutional normalization improves the robustness of common ConvNets such as ResNet and the performance of GAN. We verify our findings via extensive numerical experiments on CIFAR-10, CIFAR-100, and ImageNet.

show abstract

Finding the Sparsest Vectors in a Subspace: Theory, Algorithms, and Applications

Cited by 5 publications

References 115 publications

A Geometric Analysis of Neural Collapse with Unconstrained Features

A Geometric Analysis of Neural Collapse with Unconstrained Features

On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features

Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training

Contact Info

Product

Resources

About