Generalization Comparison of Deep Neural Networks via Output Sensitivity

Towards Understanding the Effectiveness of Attention Mechanism

“…The smooth and stable learned landscapes of AM equipped CNNs indicated they might be less sensitivity to the perturbations of input samples. It was verified by comparing the sensitivity coefficients [35], [36] between AM equipped and vanilla CNNs, and they yielded smaller sensitivity scores compared to vanilla CNNs. As various works [35], [36] demonstrated, CNNs with smaller sensitivity coefficient could generate better, which was exactly what the smooth and stable learned landscapes of AM equipped CNNs brought about.…”

“…Based on the visualization result, however, we conjectured that AM CNNs not only learned smoother and more stable curves along single-class and cross-class ellipses, but all their learned landscapes. It made AM CNNs less sensitivity to all kinds of perturbations, which brought about high generalization abilities of CNNs [35], [36]. To verify the conjecture, we compared the sensitivity scores between AM CNNs and vanilla CNNs in Section IV-B3.…”

“…The result showed that compared with the curved represented by vanilla CNNs, the curved represented by AM CNNs were smoother and more stable, which might indicate they were less sensitive to the perturbations of real samples. To verify it, two kinds of sensitivity metrics, J [36] and S [35] were further compared between AM CNNs and vanilla CNNs. It showed that with smoother and more stable landscapes learned by AM CNNs, they yielded smaller sensitivity scores and were less sensitivity to the perturbations of real samples compared to vanilla CNNs.…”

“…It showed that with smoother and more stable landscapes learned by AM CNNs, they yielded smaller sensitivity scores and were less sensitivity to the perturbations of real samples compared to vanilla CNNs. Several works [35], [36] have revealed the fact that the sensitivity of CNNs was very related to their generalization ability, i.e., CNNs with smaller sensitivity scores could generate better and yield better performance. Thus, we claimed that the high order non-linearity operator, feature map multiplication in AM brought about smooth and stable learned landscapes, which contributed to their less sensitivity to the perturbations of input samples and made they generate better than vanilla CNNs.…”

“…3) Sensitivity and Generalization: Compared to vanilla CNNs, AM equipped CNNs learned smoother and more stable curves along single-class or cross-class ellipses, which brought about their less sensitivity to perturbations containing information of samples belonging to the same or different classes. This section compared two kinds of sensitivity scores, S [35] and J [36], between vanilla CNNs and AM CNNs, which represented their sensitivity to all kinds of perturbations. It would demonstrate that with smoother and more stable learned landscapes, AM CNNs yielded smaller sensitivity scores.…”