RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning

Abstract

Semi-supervised learning (SSL) algorithms have had great success in recent years in limited labeled data regimes. However, the current state-of-the-art SSL algorithms are computationally expensive and entail significant compute time and energy requirements. This can prove to be a huge limitation for many smaller companies and academic groups. Our main insight is that training on a subset of unlabeled data instead of entire unlabeled data enables the current SSL algorithms to converge faster, thereby reducing the computational costs significantly. In this work, we propose RETRIEVE 1 , a coreset selection framework for efficient and robust semi-supervised learning. RETRIEVE selects the coreset by solving a mixed discrete-continuous bi-level optimization problem such that the selected coreset minimizes the labeled set loss. We use a one-step gradient approximation and show that the discrete optimization problem is approximately submodular, thereby enabling simple greedy algorithms to obtain the coreset. We empirically demonstrate on several real-world datasets that existing SSL algorithms like VAT, Mean-Teacher, FixMatch, when used with RETRIEVE, achieve a) faster training times, b) better performance when unlabeled data consists of Out-of-Distribution (OOD) data and imbalance. More specifically, we show that with minimal accuracy degradation, RETRIEVE achieves a speedup of around 3× in the traditional SSL setting and achieves a speedup of 5× compared to state-of-the-art (SOTA) robust SSL algorithms in the case of imbalance and OOD data.However, the current SOTA SSL algorithms are compute-intensive with very large training times. For example, from our personal experience, training a WideResNet model [52] on a CIFAR10 [24] 1 coResets for EfficienT and Robust semI-supErVised lEarning Preprint. Under review.

Data pruning and neural scaling laws: fundamental limitations of score-based algorithms

“…For instance, one could argue that preserving the gradient is an important feature to have in the coreset as it would lead to similar minima [31,32]. Other work considered the problem of corset selection as a two-stage optimization problem where the subset selection can be seen also as an optimization problem [33,34]. Other methods consider conisder the likelihood and its connection with submodular functions in order to select the subset [35,36].…”

DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning

“…Existing studies usually consider the selection of subset (optimization of samples S or selection weights w) as the outer objective and the optimization of model parameters θ on S as the inner objective. Representative methods include cardinality-constrained bilevel optimization [24] for continual learning, RETRIEVE for semisupervised learning (SSL) [25], and GLISTER [2] for supervised learning and active learning.…”

“…RETRIEVE. The RETRIEVE method [25] discusses the scenario of SSL under bilevel optimization, where we have both a labeled set T and an unlabled set P. The bilevel optimization problem in RETRIEVE is formulated as…”

“…Besides the above, coreset selection is studied and successfully applied in many other machine learning problems, such as robust learning against noise [39,25,31], clustering [8,9,40], semisupervised learning [41,25], unsupervised learning [42], efficient GAN training [15], regression tasks [19,43] etc.…”