GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning

Abstract

Large scale machine learning and deep models are extremely data-hungry. Unfortunately, obtaining large amounts of labeled data is expensive, and training state-of-the-art models (with hyperparameter tuning) requires significant computing resources and time. Secondly, real-world data is noisy and imbalanced. As a result, several recent papers try to make the training process more efficient and robust. However, most existing work either focuses on robustness or efficiency, but not both. In this work, we introduce GLISTER, a GeneraLIzation based data Subset selecTion for Efficient and Robust learning framework. We formulate GLISTER as a mixed discrete-continuous bi-level optimization problem to select a subset of the training data, which maximizes the log-likelihood on a held-out validation set. We then analyze GLISTER for simple classifiers such as gaussian and multinomial naive-bayes, k-nearest neighbor classifier, and linear regression and show connections to submodularity. Next, we propose an iterative online algorithm GLISTER-ONLINE, which performs data selection iteratively along with the parameter updates, and can be applied to any loss-based learning algorithm. We then show that for a rich class of loss functions including cross-entropy, hinge-loss, squared-loss, and logistic-loss, the inner discrete data selection is an instance of (weakly) submodular optimization, and we analyze conditions for which GLISTER-ONLINE reduces the validation loss and converges. Finally, we propose GLISTER-ACTIVE, an extension to batch active learning, and we empirically demonstrate the performance of GLISTER on a wide range of tasks including, (a) data selection to reduce training time, (b) robust learning under label noise and imbalance settings, and (c) batch-active learning with a number of deep and shallow models. We show that our framework improves upon the state of the art both in efficiency and accuracy (in cases (a) and (c)) and is more efficient compared to other state-of-the-art robust learning algorithms in case (b). The code for GLISTER is at: https://github.com/dssresearch/GLISTER.

Considering various aspects of models’ quality in the ML pipeline - application in the logistics sector

“…A combination of the above techniques would also be an exciting future research direction, primarily since many attributes are somehow related, e.g., principal components generated from the same original features or all the columns in databases referring to the same source of knowledge. Combining these techniques with stateof-the-art instance selection and training set selection methods would also be of great importance [42].…”

CLINICAL: Targeted Active Learning for Imbalanced Medical Image Classification

“…Closely related to our method Clinical, are methods which optimize an objective that involves a held-out set. GradMatch [13] uses an orthogonal matching pursuit algorithm to select a subset whose gradient closely matches the gradient of a validation set. Another method, Glister-Active [14] formulates an acquisition function that maximizes the log-likelihood on a held-out validation set.…”

“…GradMatch [13] uses an orthogonal matching pursuit algorithm to select a subset whose gradient closely matches the gradient of a validation set. Another method, Glister-Active [14] formulates an acquisition function that maximizes the log-likelihood on a held-out validation set. We adopt GradMatch and Glister-Active as baselines that targets rare classes in our class imbalance setting and refer to it T-GradMatch and T-Glister in Sec.…”

Partitioned Gradient Matching-based Data Subset Selection for Compute-Efficient Robust ASR Training

“…Thus, storing all the instance-wise loss gradients at once is not feasible for RNN-T systems. Killamsetty et al (2021a) lects mini-batches (like used in SGD) instead of individual instances. Reduction in memory by using this technique is also not much for ASR systems such as RNN-T, since batch size used here is often small.…”

“…One way to make ASR training more efficient is to train on a subset of the training data, which ensures minimum performance loss (Killamsetty et al, 2021a;Wei et al, 2014;Kaushal et al, 2019;Coleman et al, 2020;Har-Peled and Mazumdar, 2004;Clarkson, 2010;Killamsetty et al, 2021b;Liu et al, 2017). Since training on a subset reduces end-to-end time, the hyperparameter tuning time is also reduced.…”