Snapshot Distillation: Teacher-Student Optimization in One Generation

Abstract

Optimizing a deep neural network is a fundamental task in computer vision, yet direct training methods often suffer from over-fitting. Teacher-student optimization aims at providing complementary cues from a model trained previously, but these approaches are often considerably slow due to the pipeline of training a few generations in sequence, i.e., time complexity is increased by several times. This paper presents snapshot distillation (SD), the first framework which enables teacher-student optimization in one generation. The idea of SD is very simple: instead of borrowing supervision signals from previous generations, we extract such information from earlier epochs in the same generation, meanwhile make sure that the difference between teacher and student is sufficiently large so as to prevent under-fitting. To achieve this goal, we implement SD in a cyclic learning rate policy, in which the last snapshot of each cycle is used as the teacher for all iterations in the next cycle, and the teacher signal is smoothed to provide richer information. In standard image classification benchmarks such as CIFAR100 and ILSVRC2012, SD achieves consistent accuracy gain without heavy computational overheads. We also verify that models pre-trained with SD transfers well to object detection and semantic segmentation in the PascalVOC dataset.

A Generalized and Robust Method Towards Practical Gaze Estimation on Smart Phone

“…1, compared with iTracker [9], our result outperforms in both iphone dataset and ipad dataset, achieving 4.8% and 5.3% error reduction respectively. Compared with SD [26] which is reproduced by us for gaze estimation, our error is a little higher in ipad set, but lower in iphone set and the total set. iphone ipad total iTracker [9] 1.86 2.81 2.05 SD [26] 1.81 2.61 1.97 TAT 1.77 2.66 1.95 Table 1.…”

“…After that, we compare with two methods. The first one is iTracker [9], which is the state-of-the-art in 2-D gaze estimation, and the second one is SD [26], which is the state-of-the-art general knowledge distillation method. We reproduce their method for gaze estimation task.…”

“…Knowledge Distillation. Teacher-student framework has been demonstrated to improve the performance when the student has the same architecture with the teacher [26,25]. Benefiting from the soft supervision provided by the teacher, student network can learn from the inter-class similarity and potentially lower the risk of over-fitting [26].…”

“…Teacher-student framework has been demonstrated to improve the performance when the student has the same architecture with the teacher [26,25]. Benefiting from the soft supervision provided by the teacher, student network can learn from the inter-class similarity and potentially lower the risk of over-fitting [26]. To reduce the time complexity increased by educating time, Yang et al [26] present snapshot distillation, which enables teacher-student optimization in one generation.…”

“…The function of this module is to provide similarity information in the neighboring gaze positions from teacher's output iteratively. In more detail, the whole generation is split into K minigenerations following [26]. Each mini-generation has L epoches except the first one, which contains L + 1 epoches.…”