Control the number of skip‐connects to improve robustness of the NAS algorithm

Abstract

Recently, the gradient-based neural architecture search has made remarkable progress with the characteristics of high efficiency and fast convergence. However, two common problems in the gradient-based NAS algorithms are found. First, with the increase in the raining time, the NAS algorithm tends to skip-connect operation, leading to performance degradation and instability results. Second, another problem is no reasonable allocation of computing resources on valuable candidate network models. The above two points lead to the difficulty in searching the optimal sub-network and poor stability. To address them, the trick of pre-training the super-net is applied, so that each operation has an equal opportunity to develop its strength, which provides a fair competition condition for the convergence of the architecture parameters. In addition, a skip-controller is proposed to ensure each sampled sub-network with an appropriate number of skip-connects. The experiments were performed on three mainstream datasets CIFAR-10, CIFAR-100 and ImageNet, in which the improved method achieves comparable results with higher accuracy and stronger robustness.

T-TAME: Trainable Attention Mechanism for Explaining Convolutional Networks and Vision Transformers

“…Similarly, in the case of the VGG-16 backbone: the degradation of performance when masking inputs using the ViT-specific procedure is clear, although less pronounced than what it was for ResNet-50. This can be attributed to the fact that the VGG-16 architecture has no skip connections: in [59] it has been shown that limiting the number of skip connections improves robustness. Summarily, this ablation experiment demonstrates the importance of handling the perturbation of inputs in the case of CNN and ViT backbones differently, in agreement with what we proposed in Section III-C3.…”