MUSCLE: Strengthening Semi-Supervised Learning Via Concurrent Unsupervised Learning Using Mutual Information Maximization

Abstract

Deep neural networks are powerful, massively parameterized machine learning models that have been shown to perform well in supervised learning tasks. However, very large amounts of labeled data are usually needed to train deep neural networks. Several semi-supervised learning approaches have been proposed to train neural networks using smaller amounts of labeled data with a large amount of unlabeled data. The performance of these semisupervised methods significantly degrades as the size of labeled data decreases. We introduce Mutual-informationbased Unsupervised & Semi-supervised Concurrent LEarning (MUSCLE), a hybrid learning approach that uses mutual information to combine both unsupervised and semisupervised learning. MUSCLE can be used as a standalone training scheme for neural networks, and can also be incorporated into other learning approaches. We show that the proposed hybrid model outperforms state of the art on several standard benchmarks, including CIFAR-10, CIFAR-100, and Mini-Imagenet. Furthermore, the performance gain consistently increases with the reduction in the amount of labeled data, as well as in the presence of bias. We also show that MUSCLE has the potential to boost the classification performance when used in the fine-tuning phase for a model pre-trained only on unlabeled data.

Risk Bounds for Learning via Hilbert Coresets

“…The right plot in Figure 1 shows the results for the CIFAR-100 model with the state-of-the-art neural network architecture and training algorithm as in [18]. For this model, the coreset upper bound-I is nonvacuous at all training checkpoints, while the coreset upper bound-II is non-vacuous for training set sizes larger than around 8000.…”

“…The above problem is similar to the Hilbert coreset problem posed in (8) with J(w) in ( 8) replaced by a slightly different quantity J( w, p). In contrast to (8), the objective function J( w, p) in (18) depends on two vectors { w, p}. Since the goal is to provide a bound for all D ∈ D (K) CH , one can consider the following optimization problem by adapting the problem in (8) to this case: max…”

“…As a performer on DARPA Learning with Less Labeling (LwLL) program, the goal of our team was to develop novel theoretical bounds for other performers who are working on the model-building and training of new algorithms for efficient learning of classifiers with fewer labels. As part of this exercise, we approached the authors in [18] (who are part of the Information Sciences Institute (ISI) team) for the training models developed by them and developed theoretical bounds for those models (the associated hypothesis classes H). As we will show below, we obtained tight non-vacuous upper bounds for such hypothesis classes as well, indicating the power and diversity of the proposed approach.…”

“…We summarize the training algorithm used in [18] for completeness. [18] uses mutual-information based unsupervised & semi-supervised Concurrent learning (MUSCLE), a hybrid learning approach that uses mutual information to combine both unsupervised and semisupervised learning. MUSCLE can be used as a standalone training scheme for neural networks, and can also be incorporated into other learning approaches.…”

“…Furthermore, a consistency loss that measures the consistency of the prediction on an input sample and an augmented sample, can be added as a regularization term. Please refer to [18] and references therein for further details about the training algorithm and neural network models (hypothesis classes).…”