Plugin Networks for Inference under Partial Evidence

Abstract

In this paper we propose a novel method to incorporate partial evidence in the inference of deep convolutional neural networks. Contrary to the existing, top performing methods, which either iteratively modify the input of the network or exploit external label taxonomy to take the partial evidence into account, we add separate network modules ("Plugin Networks") to the intermediate layers of a pretrained convolutional network. The goal of these modules is to incorporate additional signal, i.e. information about known labels, into the inference procedure and adjust the predicted output accordingly. Since the attached plugins have a simple structure, consisting of only fully connected layers, we drastically reduced the computational cost of training and inference. At the same time, the proposed architecture allows to propagate information about known labels directly to the intermediate layers to improve the final representation. Extensive evaluation of the proposed method confirms that our Plugin Networks outperform the state-of-the-art in a variety of tasks, including scene categorization, multi-label image annotation and semantic segmentation.

Flow Plugin Network for conditional generation

“…That issue is tackled by plugin models introduced in [5]. The idea behind plugins is to incorporate an additional information, so-called partial evidence, to the trained base model without modifying the parameters.…”

“…We assume that base model has an autoencoder architecture, i.e., it has an encoder and a decoder network which enables to convert image to the latent space representation and invert that transformation respectively (see Figure 1). In our method, we utilize the concept of the plugin network [5] which task is to incorporate information known during the inference time to the pretrained model and extend that approach to generative models. As the flow plugin model, we will use the conditional normalizing flow models -Conditional Masked Autoregressive Flow and Conditional Real NVP.…”