It's All In the Teacher: Zero-Shot Quantization Brought Closer to the Teacher

Choi, Kanghyun; Lee, Hye Yoon; Hong, Deokki; Yu, Jeesuk; Park, Noseong; Kim, Youngsok; Lee, Jinho

doi:10.1109/cvpr52688.2022.00813

Cited by 25 publications

(14 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To verify the superiority of AdaSG, we compare it with typical DFQ approaches, including: GDFQ (Xu et al 2020), ARC (Zhu et al 2021) and Qimera (Choi et al 2021): reconstructing the original data from P; ZAQ (Liu, Zhang, and Wang 2021) focuses primarily on the adversarial sample generation rather than adversarial game process for AdaSG; IntraQ (Zhong et al 2022) optimizes the noise to obtain fake sample without a generator; AIT (Choi et al 2022) focuses on improving the loss function and manipulating the gradients for ARC to generate better sample, denoted as ARC+AIT.…”

Section: Comparison With State-of-the-artsmentioning

confidence: 99%

Rethinking Data-Free Quantization as a Zero-Sum Game

Bian¹,

Wang²,

Hong³

et al. 2023

Preprint

View full text Add to dashboard Cite

Data-free quantization (DFQ) recovers the performance of quantized network (Q) without accessing the real data, but generates the fake sample via a generator (G) by learning from full-precision network (P) instead. However, such sample generation process is totally independent of Q, specialized as failing to consider the adaptability of the generated samples, i.e., beneficial or adversarial, over the learning process of Q, resulting into non-ignorable performance loss. Building on this, several crucial questions -how to measure and exploit the sample adaptability to Q under varied bit-width scenarios? how to generate the samples with desirable adaptability to benefit the quantized network? -impel us to revisit DFQ. In this paper, we answer the above questions from a game-theory perspective to specialize DFQ as a zero-sum game between two players -a generator and a quantized network, and further propose an Adaptability-aware Sample Generation (AdaSG) method. Technically, AdaSG reformulates DFQ as a dynamic maximization-vs-minimization game process anchored on the sample adaptability. The maximization process aims to generate the sample with desirable adaptability, such sample adaptability is further reduced by the minimization process after calibrating Q for performance recovery. The Balance Gap is defined to guide the stationarity of the game process to maximally benefit Q. The theoretical analysis and empirical studies verify the superiority of AdaSG over the state-of-the-arts. Our code is available at https://github.com/hfutqian/AdaSG.

show abstract

Section: Comparison With State-of-the-artsmentioning

confidence: 99%

Rethinking Data-Free Quantization as a Zero-Sum Game

Bian¹,

Wang²,

Hong³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Enlightened by BSS [34], Qimera enabled the generator to synthesize boundary support samples through superposed latent embeddings, which improved the ability of the quantized model to identify the class boundary. In particular, AIT [9] paid no attention to the quality of synthetic samples, but it improved the training strategy of the quantized model.…”

Section: Related Workmentioning

confidence: 99%

“…Given the above considerations, data-free quantization has attracted much attention from researchers [8][9][10]. This technique can quantize a network precisely without accessing any authentic data.…”

Section: Introductionmentioning

confidence: 99%

“…Some researchers obtained synthetic samples that resemble the distribution of the authentic sample by using information from the pre-trained full-precision network, such as batchnormalization (BN) statistics. These approaches can be categorized into noise-optimized data-free quantization [8,[12][13][14] and generative data-free quantization [9,[15][16][17]. The former initializes a sample that satisfies the gaussian distribution, and the dimension of the sample is consistent with the size of a real sample.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Acq: Improving Generative Data-Free Quantization Via Attention Correction

Li¹,

Guo²,

Dai³

et al. 2023

Preprint

View full text Add to dashboard Cite

“…The best results are reported with boldface. and Qimera [3]: reconstructing the original data from P; ZAQ [11] focuses primarily on the adversarial sample generation rather than adversarial game process for AdaDFG; IntraQ [24] optimizes the noise to obtain fake sample without a generator; AIT [4] improves the loss function and gradients for ARC to generate better sample, denoted as ARC+AIT; AdaSG [17] focuses on the zero-sum game framework, serving as a special case of AdaDFQ. Table 2 summarizes our following findings: 1) AdaDFQ obtains a significant and consistent accuracy gain over the state-of-the-arts, in line with our purpose of optimizing the margin to generate the sample with adaptive adaptability to Q (Sec.2.3).…”

Section: Comparison With State-of-the-artsmentioning

confidence: 99%

Adaptive Data-Free Quantization

Bian¹,

Wang²,

Hong³

et al. 2023

Preprint

View full text Add to dashboard Cite

Data-free quantization (DFQ) recovers the performance of quantized network (Q) without the original data, but generates the fake sample via a generator (G) by learning from full-precision network (P), which, however, is totally independent of Q, overlooking the adaptability of the knowledge from generated samples, i.e., informative or not to the learning process of Q, resulting into the overflow of generalization error. Building on this, several critical questions -how to measure the sample adaptability to Q under varied bit-width scenarios? whether the largest adaptability is the best? how to generate the samples with adaptive adaptability to improve Q's generalization? To answer the above questions, in this paper, we propose an Adaptive Data-Free Quantization (AdaDFQ) method, which revisits DFQ from a zero-sum game perspective upon the sample adaptability between two players -a generator and a quantized network. Following this viewpoint, we further define the disagreement and agreement samples to form two boundaries, where the margin is optimized to adaptively regulate the adaptability of generated samples to Q, so as to address the over-and-under fitting issues. Our AdaDFQ reveals : 1) the largest adaptability is NOT the best for sample generation to benefit Q's generalization; 2) the knowledge of the generated sample should not be informative to Q only, but also related to the category and distribution information of the training data for P. The theoretical and empirical analysis validate the advantages of AdaDFQ over the state-of-the-arts. Our code is available at https://github.com/hfutqian/AdaDFQ.

show abstract

It's All In the Teacher: Zero-Shot Quantization Brought Closer to the Teacher

Cited by 25 publications

References 28 publications

Rethinking Data-Free Quantization as a Zero-Sum Game

Rethinking Data-Free Quantization as a Zero-Sum Game

Acq: Improving Generative Data-Free Quantization Via Attention Correction

Adaptive Data-Free Quantization

Contact Info

Product

Resources

About