Regularizing Neural Networks via Minimizing Hyperspherical Energy

Lin, Rongmei; Liu, Weiyang; Liu, Zhen; Feng, Chen; Yu, Zhiding; Rehg, James M.; Xiong, Li; Song, Le

doi:10.1109/cvpr42600.2020.00695

Cited by 16 publications

(21 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They proposed several orthogonal weight normalization methods to solve optimization over multiple dependent Stiefel manifolds. MHH-based methods (Liu et al 2018;Lin et al 2020) are inspired by the Thomson problem in physics and define the hyperspherical energy to characterize the diversity on a unit hypersphere and shows significant and consistent improvement in supervised learning tasks. Since the orthogonal regularization is too limiting (Miyato et al 2018), (Brock, Donahue, and Simonyan 2019) explored several variants designed to relax the constraint.…”

Section: Related Workmentioning

confidence: 99%

Towards an Effective Orthogonal Dictionary Convolution Strategy

Lai

et al. 2022

AAAI

View full text Add to dashboard Cite

Orthogonality regularization has proven effective in improving the precision, convergence speed and the training stability of CNNs. Here, we propose a novel Orthogonal Dictionary Convolution Strategy (ODCS) on CNNs to improve orthogonality effect by optimizing the network architecture and changing the regularized object. Specifically, we remove the nonlinear layer in typical convolution block “Conv(BN) + Nonlinear + Pointwise Conv(BN)”, and only impose orthogonal regularization on the front Conv. The structure, “Conv(BN) + Pointwise Conv(BN)”, is then equivalent to a pair of dictionary and encoding, defined in sparse dictionary learning. Thanks to the exact and efficient representation of signal with dictionaries in low-dimensional projections, our strategy could reduce the superfluous information in dictionary Conv kernels. Meanwhile, the proposed strategy relieves the too strict orthogonality regularization in training, which makes hyper-parameters tuning of model to be more flexible. In addition, our ODCS can modify the state-of-the-art models easily without any extra consumption in inference phase. We evaluate it on a variety of CNNs in small-scale (CIFAR), large-scale (ImageNet) and fine-grained (CUB-200-2011) image classification tasks, respectively. The experimental results show that our method achieve a stable and superior improvement.

show abstract

Section: Related Workmentioning

confidence: 99%

Towards an Effective Orthogonal Dictionary Convolution Strategy

Lai

et al. 2022

AAAI

View full text Add to dashboard Cite

show abstract

“…. Unrolling is shown useful in [12,41,46,47,58] and shares similar spirits with back-propagation through time in recurrent networks [85] and meta-learning [20]. The greedy policy is the optimal solution to Eq.…”

Section: Learning a Parameterized Teaching Policymentioning

confidence: 99%

Iterative Teaching by Label Synthesis

Liu

Wang

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we consider the problem of iterative machine teaching, where a teacher provides examples sequentially based on the current iterative learner. In contrast to previous methods that have to scan over the entire pool and select teaching examples from it in each iteration, we propose a label synthesis teaching framework where the teacher randomly selects input teaching examples (e.g., images) and then synthesizes suitable outputs (e.g., labels) for them. We show that this framework can avoid costly example selection while still provably achieving exponential teachability. We propose multiple novel teaching algorithms in this framework. Finally, we empirically demonstrate the value of our framework.35th Conference on Neural Information Processing Systems (NeurIPS 2021).

show abstract

“…Hyperspherical learning. Beyond face recognition, the idea of learning a representation on the hypersphere is also shown generally useful in a diverse set of applications, such as few-shot recognition [46], [47], [48], [49], [50], deep metric learning [26], [51], self-supervised learning [52], [53], [54], [55], generative models [56], [57], geometric learning [58], [59], [60], person re-identification [27], [61], [62], [63], speech processing [64], [65], [66], [67] and text processing [68]. It has been widely observed that constraining the embedding space on a hypersphere is beneficial to generalizability.…”

Section: Related Workmentioning

confidence: 99%

SphereFace Revived: Unifying Hyperspherical Face Recognition

Liu¹,

Wen²,

Raj³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

This paper addresses the deep face recognition problem under an open-set protocol, where ideal face features are expected to have smaller maximal intra-class distance than minimal inter-class distance under a suitably chosen metric space. To this end, hyperspherical face recognition, as a promising line of research, has attracted increasing attention and gradually become a major focus in face recognition research. As one of the earliest works in hyperspherical face recognition, SphereFace explicitly proposed to learn face embeddings with large inter-class angular margin. However, SphereFace still suffers from severe training instability which limits its application in practice. In order to address this problem, we introduce a unified framework to understand large angular margin in hyperspherical face recognition. Under this framework, we extend the study of SphereFace and propose an improved variant with substantially better training stability -SphereFace-R. Specifically, we propose two novel ways to implement the multiplicative margin, and study SphereFace-R under three different feature normalization schemes (no feature normalization, hard feature normalization and soft feature normalization). We also propose an implementation strategy -"characteristic gradient detachment" -to stabilize training. Extensive experiments on SphereFace-R show that it is consistently better than or competitive with state-of-the-art methods.

show abstract

Regularizing Neural Networks via Minimizing Hyperspherical Energy

Cited by 16 publications

References 26 publications

Towards an Effective Orthogonal Dictionary Convolution Strategy

Towards an Effective Orthogonal Dictionary Convolution Strategy

Iterative Teaching by Label Synthesis

SphereFace Revived: Unifying Hyperspherical Face Recognition

Contact Info

Product

Resources

About