Online Knowledge Distillation with Diverse Peers

Abstract

Distillation is an effective knowledge-transfer technique that uses predicted distributions of a powerful teacher model as soft targets to train a less-parameterized student model. A pre-trained high capacity teacher, however, is not always available. Recently proposed online variants use the aggregated intermediate predictions of multiple student models as targets to train each student model. Although group-derived targets give a good recipe for teacher-free distillation, group members are homogenized quickly with simple aggregation functions, leading to early saturated solutions. In this work, we propose Online Knowledge Distillation with Diverse peers (OKDDip), which performs two-level distillation during training with multiple auxiliary peers and one group leader. In the first-level distillation, each auxiliary peer holds an individual set of aggregation weights generated with an attention-based mechanism to derive its own targets from predictions of other auxiliary peers. Learning from distinct target distributions helps to boost peer diversity for effectiveness of group-based distillation. The second-level distillation is performed to transfer the knowledge in the ensemble of auxiliary peers further to the group leader, i.e., the model used for inference. Experimental results show that the proposed framework consistently gives better performance than state-of-the-art approaches without sacrificing training or inference complexity, demonstrating the effectiveness of the proposed two-level distillation framework.

Industrial Cyber-Physical Systems-Based Cloud IoT Edge for Federated Heterogeneous Distillation

“…However, most of them requires the networks share the same architecture or performing the same classification task [8][17] [18]. Or the teachers are modeled as different branches in a large network, which is difficult to be applied to IoT [5] [19] [20]. The BAM model [8], can perform "multiÑsingle" distillation tasks, but places limitation on network structures.…”

“…where a 1 i is the updated weight for f i and γ is the learning rate. Note that the updated logit feature z clo is related to ta i u M i"1 as the gradient computed shown in equation (19) contains a i . Minimizing L ER clo enables dynamic adaption of the weights assigned to the fog networks.…”

“…Procedure of updating the cloud network f clo is similar to the training of the cloud-to-fog distillation process with temperature T i . In each epoch, ta i u M i"1 is first updated as shown by equation (19) and (20), then θ clo is updated by minimizing L F C pf clo , tf i u M i"1 q. After the fog-to-cloud distillation, the cloud-to-fog process is performed again with smaller learning rate.…”

“…To test the ability of our HBS model dealing with homogeneous classifiers, we selected four state-of-the-art multiteacher distillation methods, including: DML [17], CL-ILR [20], ONE [5], and OKDDip [19]. We use a single pretained network as the "baseline" providing a lower bound of classification accuracy.…”

“…Note that for CL-ILR and ONE, the 3 teachers are unified as different branches on a higher capacity student model. In this case we follow the setup in [19] where the fog networks share the first few blocks and branched by the last block. The backbone architectures used in this experiment include: WRN-20-8, Resnet32, VGG16 and Resnet110.…”