Active Online Learning in the Binary Perceptron Problem

Zhou, Haijun

doi:10.1088/0253-6102/71/2/243

Cited by 5 publications

(4 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The test error for another student perceptron learning from this training set then scales as a power law with exponent −1 for such data. Such perceptrons have also been analyzed in an active learning setting where the learner is free to design any new input to be labeled [27,28], rather than choose from a fixed set of inputs, as in data-pruning. Recent work [29] has analyzed this scenario but focused on message passing algorithms that are tailored to the case of Gaussian inputs and perceptrons, and are hard to generalize to real world settings.…”

Section: Statistical Mechanics Of Perceptron Learningmentioning

confidence: 99%

Beyond neural scaling laws: beating power law scaling via data pruning

Sorscher¹,

Geirhos²,

Shekhar³

et al. 2022

Preprint

View full text Add to dashboard Cite

Widely observed neural scaling laws, in which error falls off as a power of the training set size, model size, or both, have driven substantial performance improvements in deep learning. However, these improvements through scaling alone require considerable costs in compute and energy. Here we focus on the scaling of error with dataset size and show how both in theory and practice we can break beyond power law scaling and reduce it to exponential scaling instead if we have access to a high-quality data pruning metric that ranks the order in which training examples should be discarded to achieve any pruned dataset size. We then test this new exponential scaling prediction with pruned dataset size empirically, and indeed observe better than power law scaling performance on ResNets trained on CIFAR-10, SVHN, and ImageNet. Given the importance of finding high-quality pruning metrics, we perform the first large-scale benchmarking study of ten different data pruning metrics on ImageNet. We find most existing high performing metrics scale poorly to ImageNet, while the best are computationally intensive and require labels for every image. We therefore developed a new simple, cheap and scalable self-supervised pruning metric that demonstrates comparable performance to the best supervised metrics. Overall, our work suggests that the discovery of good data-pruning metrics may provide a viable path forward to substantially improved neural scaling laws, thereby reducing the resource costs of modern deep learning.Preprint. Under review.

show abstract

Section: Statistical Mechanics Of Perceptron Learningmentioning

confidence: 99%

Beyond neural scaling laws: beating power law scaling via data pruning

Sorscher¹,

Geirhos²,

Shekhar³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…is the i-th entry of the ℓ-th composite sample vector [23,24,25,26,27]. The parameter θ µ ∈ ±1 is the sign of µ which only affects the global sign of the inferred configuration σ 0 .…”

Section: Perceptron-learningmentioning

confidence: 99%

“…We should be able to achieve almost perfect inference of σ 0 by setting X ≥ 5N [27]. The total number of sampled independent configurations is then of order O(N 2 ).…”

Section: Perceptron-learningmentioning

confidence: 99%

Circumventing spin glass traps by microcanonical spontaneous symmetry breaking

Zhou,

Liao

2020

Preprint

Self Cite

View full text Add to dashboard Cite

The planted p-spin interaction model is a paradigm of random-graph systems possessing both a ferromagnetic ground state and an intermediate spin glass phase. Conventional simulated annealing and message-passing algorithms could not reach the planted ground state but are trapped by an exponential number of spin glass states. Here we propose discontinuous microcanonical spontaneous symmetry breaking (MSSB) as a simple mechanism to circumvent all the spin glass traps. The existence of a discontinuous MSSB phase transition is confirmed by microcanonical Monte Carlo simulations. We conjecture that the planted ground state could be retrieved in polynomial time by applying machine-learning methods (such as perceptron-learning) to microcanonically sampled independent configurations. Three candidate algorithms are proposed.

show abstract

“…Active learning was previously studied in the context of the teacher-student perceptron problem. Best known is the line of work on Query by Committee [4,18,19], dealing with the membership based active learning setting, i.e. where the samples are picked one by one into the training set and can be absolutely arbitrary N -dimensional vectors.…”

Section: Definition Of the Problem And Related Workmentioning

confidence: 99%

Large deviations for the perceptron model and consequences for active learning

Cui¹,

Saglietti²,

Zdeborová³

2019

Preprint

View full text Add to dashboard Cite

Active learning is a branch of machine learning that deals with problems where unlabeled data is abundant yet obtaining labels is expensive. The learning algorithm has the possibility of querying a limited number of samples to obtain the corresponding labels, subsequently used for supervised learning. In this work, we consider the task of choosing the subset of samples to be labeled from a fixed finite pool of samples. We assume the pool of samples to be a random matrix and the ground truth labels to be generated by a single-layer teacher random neural network. We employ replica methods to analyze the large deviations for the accuracy achieved after supervised learning on a subset of the original pool. These large deviations then provide optimal achievable performance boundaries for any active learning algorithm. We show that the optimal learning performance can be efficiently approached by simple message-passing active learning algorithms. We also provide a comparison with the performance of some other popular active learning strategies.

show abstract

Active Online Learning in the Binary Perceptron Problem

Cited by 5 publications

References 51 publications

Beyond neural scaling laws: beating power law scaling via data pruning

Beyond neural scaling laws: beating power law scaling via data pruning

Circumventing spin glass traps by microcanonical spontaneous symmetry breaking

Large deviations for the perceptron model and consequences for active learning

Contact Info

Product

Resources

About