DMA Regularization: Enhancing Discriminability of Neural Networks by Decreasing the Minimal Angle

Wang, Zhennan; Xiang, Canqun; Zou, Wenbin; Xu, Chen

doi:10.1109/lsp.2020.3037512

Cited by 7 publications

(5 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Given that the angles between the feature vector and weight vectors contain abundant discriminative information [10,16,17] and adversarial attacks attack these angles, we propose a regularization term that directly encourages the weight-feature compactness, more specifically, by minimizing the angle between adversarial feature vector and the weight vector corresponding to the ground-truth label y. In addition, prior works [18] have argued strong connections between adversarial robustness and inter-class separability.…”

Section: Proposed Methodsmentioning

confidence: 99%

Improving Adversarial Robustness with Hypersphere Embedding and Angular-based Regularizations

Fakorede¹,

Nirala²,

Modeste³

et al. 2023

Preprint

View full text Add to dashboard Cite

Adversarial training (AT) methods have been found to be effective against adversarial attacks on deep neural networks. Many variants of AT have been proposed to improve its performance. Pang et al. [1] have recently shown that incorporating hypersphere embedding (HE) into the existing AT procedures enhances robustness. We observe that the existing AT procedures are not designed for the HE framework, and thus fail to adequately learn the angular discriminative information available in the HE framework. In this paper, we propose integrating HE into AT with regularization terms that exploit the rich angular information available in the HE framework. Specifically, our method, termed angular-AT, adds regularization terms to AT that explicitly enforce weight-feature compactness and inter-class separation; all expressed in terms of angular features. Experimental results show that angular-AT further improves adversarial robustness.

show abstract

Section: Proposed Methodsmentioning

confidence: 99%

Improving Adversarial Robustness with Hypersphere Embedding and Angular-based Regularizations

Fakorede¹,

Nirala²,

Modeste³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…To train our stacked generalization model, we try to reduce redundancy among multiple model predictions by applying diversity regularization (53) on the fully connected layer of MLP. This regularization ensures diverse weight vectors by maximizing the minimal pairwise angles between vectors.…”

Section: Overall Training Objectivementioning

confidence: 99%

Robust Ensemble Person Re-Identification Via Orthogonal Fusion with Occlusion Handling

Ferdous,

2023

Preprint

View full text Add to dashboard Cite

“…Recently, Zhou et al [66] extend this line of work by learning towards the largest margin with a zero-centroid regularization. Hyperspherical uniformity can also be approximated by minimizing the maximum cosine similarity between pairs of class vectors [14,34,57]. In this work, we also start from hyperspherical uniformity and deviate from current literature on one important axis: hyperspherical uniformity does not require optimization.…”

Section: Related Workmentioning

confidence: 99%

Maximum Class Separation as Inductive Bias in One Matrix

Kasarla¹,

Burghouts²,

Spengler³

et al. 2022

Preprint

View full text Add to dashboard Cite

Maximizing the separation between classes constitutes a well-known inductive bias in machine learning and a pillar of many traditional algorithms. By default, deep networks are not equipped with this inductive bias and therefore many alternative solutions have been proposed through differential optimization. Current approaches tend to optimize classification and separation jointly: aligning inputs with class vectors and separating class vectors angularly. This paper proposes a simple alternative: encoding maximum separation as an inductive bias in the network by adding one fixed matrix multiplication before computing the softmax activations. The main observation behind our approach is that separation does not require optimization but can be solved in closed-form prior to training and plugged into a network. We outline a recursive approach to obtain the matrix consisting of maximally separable vectors for any number of classes, which can be added with negligible engineering effort and computational overhead. Despite its simple nature, this one matrix multiplication provides real impact. We show that our proposal directly boosts classification, long-tailed recognition, out-of-distribution detection, and open-set recognition, from CIFAR to ImageNet. We find empirically that maximum separation works best as a fixed bias; making the matrix learnable adds nothing to the performance. The closed-form implementation and code to reproduce the experiments are on github.Preprint. Under review.

show abstract

DMA Regularization: Enhancing Discriminability of Neural Networks by Decreasing the Minimal Angle

Cited by 7 publications

References 22 publications

Improving Adversarial Robustness with Hypersphere Embedding and Angular-based Regularizations

Improving Adversarial Robustness with Hypersphere Embedding and Angular-based Regularizations

Robust Ensemble Person Re-Identification Via Orthogonal Fusion with Occlusion Handling

Maximum Class Separation as Inductive Bias in One Matrix

Contact Info

Product

Resources

About