Unifying Heterogeneous Classifiers With Distillation

Abstract

In this paper, we study the problem of unifying knowledge from a set of classifiers with different architectures and target classes into a single classifier, given only a generic set of unlabelled data. We call this problem Unifying Heterogeneous Classifiers (UHC). This problem is motivated by scenarios where data is collected from multiple sources, but the sources cannot share their data, e.g., due to privacy concerns, and only privately trained models can be shared. In addition, each source may not be able to gather data to train all classes due to data availability at each source, and may not be able to train the same classification model due to different computational resources. To tackle this problem, we propose a generalisation of knowledge distillation to merge HCs. We derive a probabilistic relation between the outputs of HCs and the probability over all classes. Based on this relation, we propose two classes of methods based on cross-entropy minimisation and matrix factorisation, which allow us to estimate soft labels over all classes from unlabelled samples and use them in lieu of ground truth labels to train a unified classifier. Our extensive experiments on ImageNet, LSUN, and Places365 datasets show that our approaches significantly outperform a naive extension of distillation and can achieve almost the same accuracy as classifiers that are trained in a centralised, supervised manner.

Industrial Cyber-Physical Systems-Based Cloud IoT Edge for Federated Heterogeneous Distillation

“…Similar limits can be found in recent classifier amalgamation works 1 . A few recent works [21] [22] [23] has been proposed to unifying heterogeneous teacher classifiers. Without a predefined dustbin class, [23] requires overlapped classes of objects recognized by teacher models, otherwise the model failed to find an optimal feature 1 A detailed comparison at https://github.com/zju-vipa/KamalEngine alignments.…”

“…where q clo pc k P C´iq is estimated by the sum of the dustbin probabilities. As discussed in [21], grouping classes in C´i into a single dustbin classes imposes design constraint on f i . We argue that the presence of dustbin facilitates efficient implementation of the distillation loss function.…”

“…The Resnet32 teacher classify 30 classes of images, without any overlapping with the targets of the other two teachers. The two baselines we used that can deal with heterogeneous teachers with totally different architectures are: common feature learning (CFL) [23] model and the Unifying Heterogeneous Classifiers (UHC) [21] model. Both models can also used for homogeneous classifiers, but in our experiments they did not generate impressive results in the homogeneous experiment shown above.…”

Adaptive multi-teacher multi-level knowledge distillation

“…[24] applies multi-teacher learning into multi-task learning where each teacher corresponds to a task. Similarly, [26] trains a classifier in each source and unifying their classifications on an integrated label space. Nevertheless, the multi-domain or multi-task setting limits the application of the method to more general scenarios for a single task in a single domain.…”

“…Some recent studies [28,6] have taken a step along this direction by either simply treating different teachers equally or manually tuning the teacher importance weights, lacking the ability of automatically discriminating different teachers. Some other studies [19,24,26] have applied multiple teacher networks to multi-task or multidomain settings. They assume each task or domain corresponds to a teacher network.…”