DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning

Guo, Chengcheng; Zhao, Bo; Bai, Yanping

doi:10.48550/arxiv.2204.08499

Cited by 2 publications

(7 citation statements)

References 24 publications

(45 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, the ignored region can be made arbitrarily large for small enough compression levels. Therefore, we expect that the generalization performance will be affected and that the drop in performance will be amplified with smaller compression levels, regardless of the sample size n. This hypothesis is empirically validated (see [1] and Section 5).…”

Section: Asymptotic Behavior Of Sbpamentioning

confidence: 69%

“…In this regime, it has been observed (see e.g. [1]) that most SBPA algorithms underperform random pruning (randomly selected subset) 2 . To understand why this occurs, we analyze the asymptotic behavior of SBPA algorithms and identify some of their properties, particularly in the high compression level regime.…”

Section: Connection With Neural Scaling Lawsmentioning

confidence: 99%

“…We consider two image datasets: CIFAR10 with ResNet18 and CIFAR100 with ResNet34. More datasets and neural architectures are available in our code, which is based on that of [1]. The code to reproduce all our experiments will be soon open-sourced.…”

Section: Effect Of the Calibration Protocolsmentioning

confidence: 99%

“…We train all models using SGD with a decaying learning rate schedule that was empirically selected following a grid search. This learning rate schedule was also used in [1]. More details are provided in Appendix C.…”

Section: Effect Of the Calibration Protocolsmentioning

confidence: 99%

See 3 more Smart Citations

Data pruning and neural scaling laws: fundamental limitations of score-based algorithms

Ayed¹,

Hayou²

2023

Preprint

View full text Add to dashboard Cite

Data pruning algorithms are commonly used to reduce the memory and computational cost of the optimization process. Recent empirical results ([1]) reveal that random data pruning remains a strong baseline and outperforms most existing data pruning methods in the high compression regime, i.e. where a fraction of 30% or less of the data is kept. This regime has recently attracted a lot of interest as result of the role of data pruning in improving the so-called neural scaling laws; see [2], where the authors showed the need for high-quality data pruning algorithms in order to beat the sample power law. In this work, we focus on score-based data pruning algorithms and show theoretically and empirically why such algorithms fail in the high compression regime. We demonstrate "No Free Lunch" theorems for data pruning and present calibration protocols that enhance the performance of existing pruning algorithms in this high compression regime using randomization.

show abstract

Section: Asymptotic Behavior Of Sbpamentioning

confidence: 69%

Section: Connection With Neural Scaling Lawsmentioning

confidence: 99%

Section: Effect Of the Calibration Protocolsmentioning

confidence: 99%

Section: Effect Of the Calibration Protocolsmentioning

confidence: 99%

See 2 more Smart Citations

Data pruning and neural scaling laws: fundamental limitations of score-based algorithms

Ayed¹,

Hayou²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Coreset-based methods select a certain proportion of data based on certain metrics [5,15]. Lapedriza et al measure the importance of the sample by the benefits obtained from training the model on the sample [24].…”

Section: Dataset Distillationmentioning

confidence: 99%

DREAM: Efficient Dataset Distillation by Representative Matching

Liu¹,

Gu²,

Wang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Dataset distillation aims to generate small datasets with little information loss as large-scale datasets for reducing storage and training costs. Recent state-of-the-art methods mainly constrain the sample generation process by matching synthetic images and the original ones regarding gradients, embedding distributions, or training trajectories. Although there are various matching objectives, currently the method for selecting original images is limited to naive random sampling. We argue that random sampling inevitably involves samples near the decision boundaries, which may provide large or noisy matching targets. Besides, random sampling cannot guarantee the evenness and diversity of the sample distribution. These factors together lead to large optimization oscillations and degrade the matching efficiency. Accordingly, we propose a novel matching strategy named as Dataset distillation by REpresentAtive Matching (DREAM), where only representative original images are selected for matching. DREAM is able to be easily plugged into popular dataset distillation frameworks and reduce the matching iterations by 10 times without performance drop. Given sufficient training time, DREAM further provides significant improvements and achieves state-of-the-art performances.* Equal contribution. This work was partially done when Yangqing was an undergraduate intern at NUS. † project lead. 0 10 20 30 40 *UDGLHQW1RUP/ 0 50 100 150 200 250 6DPSOH1XPEHU 5DQGRPVDPSOHV '5($0VDPSOHV (a) The gradient norm distribution of the plane class in CIFAR10. Matched original images Synthetic images DREAM Random Sampling (b) The oscillation of synthetic samples during training.

show abstract

DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning

Cited by 2 publications

References 24 publications

Data pruning and neural scaling laws: fundamental limitations of score-based algorithms

Data pruning and neural scaling laws: fundamental limitations of score-based algorithms

DREAM: Efficient Dataset Distillation by Representative Matching

Contact Info

Product

Resources

About