DyRep: Bootstrapping Training with Dynamic Re-parameterization

Huang, T.; You, Suya; Zhang, Bohan; Du, Yukai; Wang, Fei; Chen, Qian; Chen, Xu

doi:10.48550/arxiv.2203.12868

Cited by 3 publications

(2 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, the performance of re-parameterization models is closely related to the type of re-parameterization operations. Some previous work [28,29] search the best combination of re-parameterization operations automatically, but the type of re-parameterization operations limits the upper limit of network performance. Therefore, the performance of model can be further improved by exploring more re-parameterization operations.…”

Section: Introductionmentioning

confidence: 99%

Efficient Re-parameterization Operations Search for Easy-to-Deploy Network Based on Directional Evolutionary Strategy

Wang

Rong

et al. 2023

Neural Process Lett

View full text Add to dashboard Cite

Structural re-parameterization (Rep) methods has achieved significant performance improvement on traditional convolutional network. Most current Rep methods rely on prior knowledge to select the reparameterization operations. However, the performance of architecture is limited by the type of operations and prior knowledge. To break this restriction, in this work, an improved re-parameterization search space is designed, which including more type of re-parameterization operations. Concretely, the performance of convolutional networks can be further improved by the search space. To effectively explore this search space, an automatic re-parameterization enhancement strategy is designed based on neural architecture search (NAS), which can search a excellent re-parameterization architecture. Besides, we visualize the output features of the architecture to analyze the reasons for the formation of the re-parameterization architecture. On public datasets, we achieve better results. Under the same training conditions as ResNet, we improve the accuracy of ResNet-50 by 1.82% on ImageNet-1k.

show abstract

Section: Introductionmentioning

confidence: 99%

Efficient Re-parameterization Operations Search for Easy-to-Deploy Network Based on Directional Evolutionary Strategy

Wang

Rong

et al. 2023

Neural Process Lett

View full text Add to dashboard Cite

show abstract

“…The advent of automatic feature engineering fuels deep neural networks to achieve remarkable success in a plethora of computer vision tasks, such as image classification [16,18,37,47,52], object detection [2,22], and semantic segmentation [5,53]. In the path of pursuing better performance, current deep learning models generally grow deeper and wider [12,44].…”

Section: Introductionmentioning

confidence: 99%

Knowledge Distillation from A Stronger Teacher

Huang¹,

You²,

Wang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Unlike existing knowledge distillation methods focus on the baseline settings, where the teacher models and training strategies are not that strong and competing as state-of-the-art approaches, this paper presents a method dubbed DIST to distill better from a stronger teacher. We empirically find that the discrepancy of predictions between the student and a stronger teacher may tend to be fairly severer. As a result, the exact match of predictions in KL divergence would disturb the training and make existing methods perform poorly. In this paper, we show that simply preserving the relations between the predictions of teacher and student would suffice, and propose a correlation-based loss to capture the intrinsic inter-class relations from the teacher explicitly. Besides, considering that different instances have different semantic similarities to each class, we also extend this relational match to the intra-class level. Our method is simple yet practical, and extensive experiments demonstrate that it adapts well to various architectures, model sizes and training strategies, and can achieve state-of-the-art performance consistently on image classification, object detection, and semantic segmentation tasks. Code is available at: https://github.com/hunto/DIST_KD.

show abstract

Efficient Re-parameterization Operations Search for Easy-to-Deploy Network Based on Directional Evolutionary Strategy

Wang

Rong

et al. 2022

Preprint

View full text Add to dashboard Cite

Structural re-parameterization (Rep) methods has achieved signifificant performance improvement on traditional convolutional network. Most current Rep methods rely on prior knowledge to select the re-parameterization operations. However, the performance of architecture is limited by the type of operations and prior knowledge. To break this restriction, in this work, an improved re-parameterization search space is designed, which including more type of re-parameterization operations. Concretely, the performance of convolutional networks can be further improved by the search space. To effectively explore this search space, an automatic re-parameterization enhancement strategy is designed based on neural architecture search (NAS), which can search a excellent re-parameterization architecture. Besides, we visualize the output features of the architecture to analyze the reasons for the formation of the re-parameterization architecture. On public datasets, we achieve better results. Under the same training conditions as ResNet, we improve the accuracy of ResNet-50 by 1.82% on ImageNet-1k.

show abstract

DyRep: Bootstrapping Training with Dynamic Re-parameterization

Cited by 3 publications

References 23 publications

Efficient Re-parameterization Operations Search for Easy-to-Deploy Network Based on Directional Evolutionary Strategy

Efficient Re-parameterization Operations Search for Easy-to-Deploy Network Based on Directional Evolutionary Strategy

Knowledge Distillation from A Stronger Teacher

Efficient Re-parameterization Operations Search for Easy-to-Deploy Network Based on Directional Evolutionary Strategy

Contact Info

Product

Resources

About