A Geometric Analysis of Neural Collapse with Unconstrained Features

Zhu, Zhihui; Ding, Tianyu; Zhou, Jinxin; Li, Xiao; You, Chong; Sulam, Jeremias; Qu, Qing

doi:10.48550/arxiv.2105.02375

Cited by 5 publications

(18 citation statements)

References 63 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Relationship With Other Results On Neural Collapsementioning

confidence: 99%

“…The most relevant study is a concurrent work [50], which provides a landscape analysis of the regularized unconstrained feature model. Zhu et al [50] turns the feature norm constraint in Fang et al [7] into feature norm regularization and still preserves the neural collapse global optimum. At the same time, it shows that the modified regularized objective shares a benign landscape, where all the critical points are strict saddles except for the global one.…”

Section: Relationship With Other Results On Neural Collapsementioning

confidence: 99%

“…Feature Norm Regularization Loss Function [33] Empirical Results Cross-Entropy Loss [45,27,7] Global Optimum Cross-Entropy Loss [29,34,13] Modified Training Dynamics 2 Loss [50] Landscape Analysis Cross-Entropy Loss…”

Section: Reference Contribution Feature Norm Constraintmentioning

confidence: 99%

“…Since the general analysis of the highly non-smooth and non-convex neural network is difficult, we peel down the last layer of the neural network and propose the following unconstrained layer-peeled model (ULPM) as a simplification to capture the main characters related to neural collapse during training dynamics. A similar simplification is commonly used in previous theoretical works [27,7,45,50], but ours does not have any constraint or regularization on features and is closer to realistic neural network models. It should be mentioned that although [29,13] also studied the unconstrained model, their analysis adopted an approximation to real dynamics and was highly dependent on the 2 loss function, which is rarely used in classification tasks.…”

Section: Problem Setupmentioning

confidence: 99%

“…The intuition behind the LPM is that modern deep networks are often highly over-parameterized, with the capacity to learn any representations of the input data. It has been shown that an equiangular tight frame (ETF), i.e., feature with neural collapse, is the only global optimum of the LPM objective (1.1) [7,27,45,50]. However, feature constraints in LPMs are unrealistic settings that are never used in practice.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

An Unconstrained Layer-Peeled Perspective on Neural Collapse

Ji¹,

Lu²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Neural collapse is a highly symmetric geometric pattern of neural networks that emerges during the terminal phase of training, with profound implications on the generalization performance and robustness of the trained networks. To understand how the last-layer features and classifiers exhibit this recently discovered implicit bias, in this paper, we introduce a surrogate model called the unconstrained layer-peeled model (ULPM). We prove that gradient flow on this model converges to critical points of a minimum-norm separation problem exhibiting neural collapse in its global minimizer. Moreover, we show that the ULPM with the cross-entropy loss has a benign global landscape for its loss function, which allows us to prove that all the critical points are strict saddle points except the global minimizers that exhibit the neural collapse phenomenon. Empirically, we show that our results also hold during the training of neural networks in real-world tasks when explicit regularization or weight decay is not used.

show abstract

Section: Relationship With Other Results On Neural Collapsementioning

confidence: 99%

Section: Relationship With Other Results On Neural Collapsementioning

confidence: 99%

Section: Reference Contribution Feature Norm Constraintmentioning

confidence: 99%

Section: Problem Setupmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

An Unconstrained Layer-Peeled Perspective on Neural Collapse

Ji¹,

Lu²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

Revealing hidden patterns in deep neural network feature space continuum via manifold learning

Islam,

Zhou,

Ren

et al. 2023

Nat Commun

View full text Add to dashboard Cite

Deep neural networks (DNNs) extract thousands to millions of task-specific features during model training for inference and decision-making. While visualizing these features is critical for comprehending the learning process and improving the performance of the DNNs, existing visualization techniques work only for classification tasks. For regressions, the feature points lie on a high dimensional continuum having an inherently complex shape, making a meaningful visualization of the features intractable. Given that the majority of deep learning applications are regression-oriented, developing a conceptual framework and computational method to reliably visualize the regression features is of great significance. Here, we introduce a manifold discovery and analysis (MDA) method for DNN feature visualization, which involves learning the manifold topology associated with the output and target labels of a DNN. MDA leverages the acquired topological information to preserve the local geometry of the feature space manifold and provides insightful visualizations of the DNN features, highlighting the appropriateness, generalizability, and adversarial robustness of a DNN. The performance and advantages of the MDA approach compared to the existing methods are demonstrated in different deep learning applications.

show abstract

Structural degeneracy and formation of crystallographic domains in epitaxial LaFeO3 films revealed by machine-learning assisted 4D-STEM

Zhu,

Lanier,

Flores

et al. 2024

Sci Rep

View full text Add to dashboard Cite

Structural domains and domain walls, inherent in single crystalline perovskite oxides, can significantly influence the properties of the material and therefore must be considered as a vital part of the design of the epitaxial oxide thin films. We employ 4D-STEM combined with machine learning (ML) to comprehensively characterize domain structures at both high spatial resolution and over a significant spatial extent. Using orthorhombic LaFeO3 as a model system, we explore the application of unsupervised and supervised ML in domain mapping, which demonstrates robustness against experiment uncertainties. The results reveal the consequential formation of multiple domains due to the structural degeneracy when LaFeO3 film is grown on cubic SrTiO3. In situ annealing of the film shows the mechanism of domain coarsening that potentially links to phase transition of LaFeO3 at high temperatures. Moreover, synthesis of LaFeO3 on DyScO3 illustrates that a less symmetric orthorhombic substrate inhibits the formation of domain walls, thereby contributing to the mitigation of structural degeneracy. High fidelity of our approach also highlights the potential for the domain mapping of other complicated materials and thin films.

show abstract

A Geometric Analysis of Neural Collapse with Unconstrained Features

Cited by 5 publications

References 63 publications

An Unconstrained Layer-Peeled Perspective on Neural Collapse

An Unconstrained Layer-Peeled Perspective on Neural Collapse

Revealing hidden patterns in deep neural network feature space continuum via manifold learning

Structural degeneracy and formation of crystallographic domains in epitaxial LaFeO3 films revealed by machine-learning assisted 4D-STEM

Contact Info

Product

Resources

About