Generative Adversarial Training for Supervised and Semi-supervised Learning

Abstract

Neural networks have played critical roles in many research fields. The recently proposed adversarial training (AT) can improve the generalization ability of neural networks by adding intentional perturbations in the training process, but sometimes still fail to generate worst-case perturbations, thus resulting in limited improvement. Instead of designing a specific smoothness function and seeking an approximate solution used in existing AT methods, we propose a new training methodology, named Generative AT (GAT) in this article, for supervised and semi-supervised learning. The key idea of GAT is to formulate the learning task as a minimax game, in which the perturbation generator aims to yield the worst-case perturbations that maximize the deviation of output distribution, while the target classifier is to minimize the impact of this perturbation and prediction error. To solve this minimax optimization problem, a new adversarial loss function is constructed based on the cross-entropy measure. As a result, the smoothness and confidence of the model are both greatly improved. Moreover, we develop a trajectory-preserving-based alternating update strategy to enable the stable training of GAT. Numerous experiments conducted on benchmark datasets clearly demonstrate that the proposed GAT significantly outperforms the state-of-the-art AT methods in terms of supervised and semi-supervised learning tasks, especially when the number of labeled examples is rather small in semi-supervised learning.

Anomaly detection using deep learning based model with feature attention

“…Generative adversarial networks (GANs) are popular in computer vision [23], [24] and anomaly detection [25] because they can generate data and handle complex data distributions effectively. GANs have a lot of benefits, but they're hard to train [26].…”

Contrastive distortion‐level learning‐based no‐reference image‐quality assessment

“…As a kind of unsupervised learning paradigm, contrastive learning directly uses the data itself as supervised information to learn the feature expression of sample data and use it for downstream tasks without relying on manually annotated category label information. 58 The basic idea of contrastive learning is shown in Figure 3. Contrastive learning learns the feature representation of samples by comparing the training data with positive and negative samples in the feature space.…”

Task‐aware swapping for efficient DNN inference on DRAM‐constrained edge systems

“…While these works can maintain high recognition accuracy, deploying a highperformance lightweight neural network requires additional efforts of manual tuning by designers, which limits their popularization. 27 Different from prior network pruning and tuning works, this paper is orthogonal to existing memory-save schemes for object detection tasks and it can be implemented as an add-on module to the existing solutions. Since the remaindered networks after pruning can be considered as an independent network for further swapping, which improves the reusability of each branch of the network.…”

“…Howard et al 26 designed two super parameters to build a small, low latency model mobilenets, to match the requirements of embedded system visual applications. While these works can maintain high recognition accuracy, deploying a high‐performance lightweight neural network requires additional efforts of manual tuning by designers, which limits their popularization 27 …”