Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction

Abstract

To learn intrinsic low-dimensional structures from high-dimensional data that most discriminate between classes, we propose the principle of Maximal Coding Rate Reduction (MCR 2 ), an information-theoretic measure that maximizes the coding rate difference between the whole dataset and the sum of each individual class. We clarify its relationships with most existing frameworks such as cross-entropy, information bottleneck, information gain, contractive and contrastive learning, and provide theoretical guarantees for learning diverse and discriminative features. The coding rate can be accurately computed from finite samples of degenerate subspace-like distributions and can learn intrinsic representations in supervised, self-supervised, and unsupervised settings in a unified manner. Empirically, the representations learned using this principle alone are significantly more robust to label corruptions in classification than those using cross-entropy, and can lead to state-of-the-art results in clustering mixed data from self-learned invariant features.

A Geometric Analysis of Neural Collapse with Unconstrained Features

“…Indeed, the study of the interplay of the learned features and the final classifier enables precise characterization of the adversarial robustness of the learned model [112]. On the other hand, another line of recent work [33,34] empirically showed and argued that mapping each class to a linearly separable subspace with maximum dimension (instead of collapsing them to a vertex of Simplex ETF) can improve robustness against random data corruptions such as label noise. Further empirical and theoretical investigations are needed to clarify our understandings of potential benefits and full implications of N C for robustness.…”

“…In addition, the variability collapse of N C aligns with information bottleneck theory [32] which hypothesizes that neural networks seek to preserve only the minimal set of information in the learned feature representations for predicting the label hence discourage any additional variabilities. On the other hand, a recent line of work [33,34] raises controversial questions on whether N C improves robustness against data corruptions by showing that diverse features that preserve the intrinsic structure of data can better handle label corruptions. Therefore, an in-depth theoretical study of the N C phenomenon could provide further insights for addressing all these fundamental questions (see Section 5 for a detailed discussion).…”

“…We note that the principles of residual learning and isometric learning, which employs an identity and isometric transformation that preserves distances (and hence, the structure of the data) at each layer, is at odds with N C, which neglects intrinsic data structures and compresses each class into a vector of Simplex ETF. Resolving this conflict may require us to rethink the cross-entropy loss as a surrogate for the risk objective and design new objective functions that respect the intrinsic structure of the data [33].…”

Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold

“…• Feature normalization is a common practice in training deep networks. Recently, many existing results demonstrated that training with feature normalization often improves the quality of learned representation with better class separation [5,13,21,42,60,73,75,81]. Such a representation is closely related to the discriminative representation in literature; see, e.g., [42,60,73,80].…”

Toward Better Generalization Bounds with Locally Elastic Stability

“…On the other hand, different mathematical models are proposed to model the structure of real-world data, such as the long tail theory (Zhu et al, 2014;Feldman, 2020;Feldman & Zhang, 2020), and the hidden manifold model (Goldt et al, 2019). Lately, Yu et al (2020) unveil several profound geometric properties of neural networks in the feature space from the viewpoint of discriminative representations (see also ; Anonymous (2021)).…”