Understanding Why Generalized Reweighting Does Not Improve Over ERM

Zhai, Runtian; Chen, Dan; Kolter, Zico; Ravikumar, Pradeep

doi:10.48550/arxiv.2201.12293

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Reweighting Does Not Improve Over ERM [Zhai et al, 2022] Summary of the Paper: The paper studies the performance of generalized reweighting (GRW) algorithms, a class of methods that aim to address distributional shift in machine learning tasks. The authors first prove that for linear models and sufficiently wide fully-connected neural networks, the implicit bias of GRW is equivalent to empirical risk minimization (ERM) when trained for an infinitely long time, and that regularization must be large enough to significantly lower the training performance in order to affect this implicit bias.…”

Section: Understanding Why Generalizedmentioning

confidence: 99%

A day in the life of ChatGPT as an academic reviewer: Investigating the potential of large language model for scientific literature review

Srivastava

2023

Preprint

View full text Add to dashboard Cite

In this paper, we investigate the potential of large language model for scientific literature review. The exponential growth of research papers is placing an increasing burden on human reviewers, making it challenging to maintain efficient and reliable review processes. To address this, we explore the use of ChatGPT to assist with the review process. Our experiments demonstrate that ChatGPT can review the papers followed by the sentiment analysis of the review of research papers and provide insights into their potential for acceptance or rejection. Although our study is limited to a small sample of papers, the results are promising and suggest that further research in this area is warranted. We note that the use of large language models for scientific literature review is still in its early stages, and there are many challenges to be addressed. Nonetheless, our work highlights the potential of these models to augment the traditional peer-review process, providing a new perspective on research papers and potentially accelerating the pace of scientific discovery and innovation.

show abstract

Section: Understanding Why Generalizedmentioning

confidence: 99%

A day in the life of ChatGPT as an academic reviewer: Investigating the potential of large language model for scientific literature review

Srivastava

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Empirically, prior works [10,50] recognize that various IW methods tend to exacerbate overfitting, which leads to a diminishing effect on stochastic gradient descent (SGD) over training epochs especially when they are applied to over-parameterized neural networks (NNs). Theoretically, previous studies prove that for over-parameterized neural networks, re-weighting algorithms do not improve over ERM because their implicit biases are (almost) equivalent [51,58,62]. In addition, some prior works also point out that using conventional regularization techniques such as weight decay cannot significantly improve the performance of IW [50].…”

Section: Introductionmentioning

confidence: 99%

UMIX: Improving Importance Weighting for Subpopulation Shift via Uncertainty-Aware Mixup

Han¹,

Zhang²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Subpopulation shift wildly exists in many real-world machine learning applications, referring to the training and test distributions containing the same subpopulation groups but varying in subpopulation frequencies. Importance reweighting is a normal way to handle the subpopulation shift issue by imposing constant or adaptive sampling weights on each sample in the training dataset. However, some recent studies have recognized that most of these approaches fail to improve the performance over empirical risk minimization especially when applied to overparameterized neural networks. In this work, we propose a simple yet practical framework, called uncertainty-aware mixup (UMIX), to mitigate the overfitting issue in over-parameterized models by reweighting the "mixed" samples according to the sample uncertainty. The training-trajectories-based uncertainty estimation is equipped in the proposed UMIX for each sample to flexibly characterize the subpopulation distribution. We also provide insightful theoretical analysis to verify that UMIX achieves better generalization bounds over prior works. Further, we conduct extensive empirical studies across a wide range of tasks to validate the effectiveness of our method both qualitatively and quantitatively.

show abstract

Understanding Why Generalized Reweighting Does Not Improve Over ERM

Cited by 2 publications

References 0 publications

A day in the life of ChatGPT as an academic reviewer: Investigating the potential of large language model for scientific literature review

A day in the life of ChatGPT as an academic reviewer: Investigating the potential of large language model for scientific literature review

UMIX: Improving Importance Weighting for Subpopulation Shift via Uncertainty-Aware Mixup

Contact Info

Product

Resources

About