Hypernetwork-Based Augmentation

Abstract

Data augmentation is an effective technique to improve the generalization of deep neural networks. Recently, AutoAugment [1] proposed a well-designed search space and a search algorithm that automatically finds augmentation policies in a data-driven manner. However, AutoAugment is computationally intensive. In this paper, we propose an efficient gradient-based search algorithm, called Hypernetwork-Based Augmentation (HBA), which simultaneously learns model parameters and augmentation hyperparameters in a single training. Our HBA uses a hypernetwork to approximate a population-based training algorithm, which enables us to tune augmentation hyperparameters by gradient descent. Besides, we introduce a weight sharing strategy that simplifies our hypernetwork architecture and speeds up our search algorithm. We conduct experiments on CIFAR-10, CIFAR-100, SVHN, and ImageNet. Our results demonstrate that HBA is significantly faster than state-of-the-art methods while achieving competitive accuracy.

DIVA: Dataset Derivative of a Learning Task

“…AutoML has focused on achieving better results via automatic model selection [17,20] including neural architecture search (NAS) [74,19,42]. Other important AutoML topics include hyper-parameter selection [37,2] and data augmentation [14,39,12,5], which are closer to our settings of optimizing the dataset weights. Since the main signal for a model's performance is the final validation loss, which requires full optimization of the model for each evaluation, AutoML approaches often incur a steep computational costs.…”