Which Samples Should be Learned First: Easy or Hard?

Zhou, Xiaoling; Wu, Ou

doi:10.48550/arxiv.2110.05481

Cited by 3 publications

(4 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, we note that some researchers suggest that better results can be obtained by training with hard samples first [32]. These authors demonstrate this for imbalanced datasets, where they define "hard" as the same as rare examples.…”

Section: Datasupporting

confidence: 53%

See 1 more Smart Citation

General Cyclical Training of Neural Networks

Smith¹

2022

Preprint

View full text Add to dashboard Cite

This paper describes the principle of "General Cyclical Training" in machine learning, where training starts and ends with "easy training" and the "hard training" happens during the middle epochs. We propose several manifestations for training neural networks, including algorithmic examples (via hyper-parameters and loss functions), data-based examples, and model-based examples. Specifically, we introduce several novel techniques: cyclical weight decay, cyclical batch size, cyclical focal loss, cyclical softmax temperature, cyclical data augmentation, cyclical gradient clipping, and cyclical semi-supervised learning. In addition, we demonstrate that cyclical weight decay, cyclical softmax temperature, and cyclical gradient clipping (as three examples of this principle) are beneficial in the test accuracy performance of a trained model. Furthermore, we discuss model-based examples (such as pretraining and knowledge distillation) from the perspective of general cyclical training and recommend some changes to the typical training methodology. In summary, this paper defines the general cyclical training concept and discusses several specific ways in which this concept can be applied to training neural networks. In the spirit of reproducibility, the code used in our experiments is available at https://github.com/lnsmith54/CFL.

show abstract

Section: Datasupporting

confidence: 53%

“…A training sample might have distracting backgrounds, corruptions, or a poor foreground/background ratio. For these reasons and more, training samples have been divided into categories of easy/medium/hard [32,33]. This has led to the research question: Which training samples should be learned first and then in what order?…”

Section: Datamentioning

confidence: 99%

General Cyclical Training of Neural Networks

Smith¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Besides the complexity in the data, there is also the property difficulty for an ML model of learning certain things [51], so recognizing something in complex sequences can be challenging (cf. section "Complexity").…”

Section: Difficultymentioning

confidence: 99%

Corner cases in machine learning processes

Heidecker,

Bieshaar,

Sick

2024

AI Perspect. Adv.

View full text Add to dashboard Cite

Applications using machine learning (ML), such as highly autonomous driving, depend highly on the performance of the ML model. The data amount and quality used for model training and validation are crucial. If the model cannot detect and interpret a new, rare, or perhaps dangerous situation, often referred to as a corner case, we will likely blame the data for not being good enough or too small in number. However, the implemented ML model and its associated architecture also influence the behavior. Therefore, the occurrence of prediction errors resulting from the ML model itself is not surprising. This work addresses a corner case definition from an ML model’s perspective to determine which aspects must be considered. To achieve this goal, we present an overview of properties for corner cases that are beneficial for the description, explanation, reproduction, or synthetic generation of corner cases. To define ML corner cases, we review different considerations in the literature and summarize them in a general description and mathematical formulation, whereby the expected relevance-weighted loss is the key to distinguishing corner cases from common data. Moreover, we show how to operationalize the corner case characteristics to determine the value of a corner case. To conclude, we present the extended taxonomy for ML corner cases by adding the input, model, and deployment levels, considering the influence of the corner case properties.

show abstract

“…Treating each training sample unequally improves the learning performance. Two cues are typically considered in designing the weighting schemes of training samples [1]. The first cue is the application context of learning tasks.…”

Section: Introductionmentioning

confidence: 99%

Understanding Difficulty-based Sample Weighting with a Universal Difficulty Measure

Zhou¹,

Ou²,

Zhu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Sample weighting is widely used in deep learning. A large number of weighting methods essentially utilize the learning difficulty of training samples to calculate their weights. In this study, this scheme is called difficulty-based weighting. Two important issues arise when explaining this scheme. First, a unified difficulty measure that can be theoretically guaranteed for training samples does not exist. The learning difficulties of the samples are determined by multiple factors including noise level, imbalance degree, margin, and uncertainty. Nevertheless, existing measures only consider a single factor or in part, but not in their entirety. Second, a comprehensive theoretical explanation is lacking with respect to demonstrating why difficulty-based weighting schemes are effective in deep learning. In this study, we theoretically prove that the generalization error of a sample can be used as a universal difficulty measure. Furthermore, we provide formal theoretical justifications on the role of difficulty-based weighting for deep learning, consequently revealing its positive influences on both the optimization dynamics and generalization performance of deep models, which is instructive to existing weighting schemes.

show abstract

Which Samples Should be Learned First: Easy or Hard?

Cited by 3 publications

References 43 publications

General Cyclical Training of Neural Networks

General Cyclical Training of Neural Networks

Corner cases in machine learning processes

Understanding Difficulty-based Sample Weighting with a Universal Difficulty Measure

Contact Info

Product

Resources

About