Nearest Class-Center Simplification through Intermediate Layers

Ido, Ben-Shaul,; Dekel, Shai

doi:10.48550/arxiv.2201.08924

Cited by 1 publication

(1 citation statement)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The work [19] shows that N C also happens on test data drawn from the same distribution asymptotically, but less collapse for finite samples [29]. Other works [29,49] demonstrated that the variability collapse of features is actually happening progressively from shallow to deep layers, and [2] showed that test performance can be improved when enforcing variability collapse on features of intermediate layers. The works [78,79] showed that fixing the classifier as a simplex ETF improves test performance on imbalanced training data and long-tailed classification problems.…”

Section: Motivations and Contributionsmentioning

confidence: 99%

Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold

Yaras¹,

Wang²,

Zhu³

et al. 2022

Preprint

View full text Add to dashboard Cite

When training overparameterized deep networks for classification tasks, it has been widely observed that the learned features exhibit a so-called "neural collapse" phenomenon. More specifically, for the output features of the penultimate layer, for each class the within-class features converge to their means, and the means of different classes exhibit a certain tight frame structure, which is also aligned with the last layer's classifier. As feature normalization in the last layer becomes a common practice in modern representation learning, in this work we theoretically justify the neural collapse phenomenon for normalized features. Based on an unconstrained feature model, we simplify the empirical loss function in a multi-class classification task into a nonconvex optimization problem over the Riemannian manifold by constraining all features and classifiers over the sphere. In this context, we analyze the nonconvex landscape of the Riemannian optimization problem over the product of spheres, showing a benign global landscape in the sense that the only global minimizers are the neural collapse solutions while all other critical points are strict saddles with negative curvature. Experimental results on practical deep networks corroborate our theory and demonstrate that better representations can be learned faster via feature normalization.

show abstract