Fast rates for noisy interpolation require rethinking the effects of inductive bias

Donhauser, Konstantin; Ruggeri, Nicolo; Stojanovic, Stefan; Yang, Fanny

doi:10.48550/arxiv.2203.03597

Cited by 1 publication

(1 citation statement)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Because of its simplicity and well-develped theory in classical machine learning [58,15,16], sparse modeling is often used to provide theoretical understanding of modern large and over-parameterized models. This include works on implicit regularization [59,60,61,62,63], nonconvex optimization [64,65], noise interpolators [66,67,68], etc. However, the aforementioned work uses sparsity as a testbed or toy model to gain insights, without implication of existence of sparsity in DNNs.…”

Section: Sparsity For Robustnessmentioning

confidence: 99%

Large Models are Parsimonious Learners: Activation Sparsity in Trained Transformers

Li¹,

You²,

Bhojanapalli³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper studies the curious phenomenon for machine learning models with Transformer architectures that their activation maps are sparse. By activation map we refer to the intermediate output of the multi-layer perceptrons (MLPs) after a ReLU activation function, and by "sparse" we mean that on average very few entries (e.g., 3.0% for T5-Base and 6.3% for ViT-B16) are nonzero for each input to MLP. Moreover, larger Transformers with more layers and wider MLP hidden dimensions are sparser as measured by the percentage of nonzero entries. Through extensive experiments we demonstrate that the emergence of sparsity is a prevalent phenomenon that occurs for both natural language processing and vision tasks, on both training and evaluation data, for Transformers of various configurations, at layers of all depth levels, as well as for other architectures including MLP-mixers and 2-layer MLPs. We show that sparsity also emerges using training datasets with random labels, or with random inputs, or with infinite amount of data, demonstrating that sparsity is not a result of a specific family of datasets. We discuss how sparsity immediately implies a way to significantly reduce the FLOP count and improve efficiency for Transformers. Moreover, we demonstrate perhaps surprisingly that enforcing an even sparser activation via Top-k thresholding with a small value of k brings a collection of desired but missing properties for Transformers, namely less sensitivity to noisy training data, more robustness to input corruptions, and better calibration for their prediction confidence.

show abstract