Write a Classifier: Predicting Visual Classifiers from Unstructured Text

Abstract

Abstract-People typically learn through exposure to visual concepts associated with linguistic descriptions. For instance, teaching visual object categories to children is often accompanied by descriptions in text or speech. In a machine learning context, these observations motivates us to ask whether this learning process could be computationally modeled to learn visual classifiers. More specifically, the main question of this work is how to utilize purely textual description of visual classes with no training images, to learn explicit visual classifiers for them. We propose and investigate two baseline formulations, based on regression and domain transfer, that predict a linear classifier. Then, we propose a new constrained optimization formulation that combines a regression function and a knowledge transfer function with additional constraints to predict the parameters of a linear classifier. We also propose a generic kernelized models where a kernel classifier is predicted in the form defined by the representer theorem. The kernelized models allow defining and utilizing any two RKHS 1 kernel functions in the visual space and text space, respectively. We finally propose a kernel function between unstructured text descriptions that builds on distributional semantics, which shows an advantage in our setting and could be useful for other applications. We applied all the studied models to predict visual classifiers on two fine-grained and challenging categorization datasets (CU Birds and Flower Datasets), and the results indicate successful predictions of our final model over several baselines that we designed.

Link the Head to the "Beak": Zero Shot Learning from Noisy Text Description at Part Precision

“…Baselines and Competing Methods: The performance of our approach is compared to six state-of-the-art algorithms: SJE [6], MCZSL [3], ZSLNS [36], ESZSL [39], WAC [14]. The source code of ESZSL and ZSLNS are available online, and we get the code of WAC [14,13] from its author. For MCZSL and SJE, since their source codes are not available, we directly copy the highest scores for non-attribute settings reported in [3,6].…”

“…Note that some of these methods were applied on attributes prediction (e.g., ZSLNS [36], SynC [10], ESZSL [39] ) or image-sentence similarity (e.g.,Order Embedding [46]). We used the publicly available code of these methods and other text-based methods like (ZSLNS [36], WAC [14], WAC-kernel [13]) to apply them on our setting. Note that the conventional split setting for zero-shot learning is Super-Category Shared splitting (i.e.…”

“…WAC-Linear [14] 27.0 5.0 WAC-Kernel [13] 33.5 7.7 ESZSL [39] 28.5 7.4 SJE [6] 29.9 -ZSLNS [36] 29.1 7.3 SynC f ast [10] 28.0 8.6 SynC OV O [10] 12.5 5.9 Order Embedding [46] 17.3 5.9 Ours-DET 37.2 9.7 Generalized Zero-Shot Learning Performance. The conventional zero-shot learning that we discussed earlier classifies test examples into unseen classes without considering the seen classes in test phase.…”

Choose Your Neuron: Incorporating Domain Knowledge Through Neuron-Importance

“…[18]). The proposed approaches mainly rely on learning a similarity function between text descriptions and images (either linearly [8,25] or non-linearly via deep neural networks [18] or kernels [7]). At test-time, classification is performed by associating the image to the class with the highest similarity to the corresponding class-level text.…”

“…Many existing zero-shot learning approaches make use of deep features (i.e. vectors of activations from some late layer in a network pretrained on some largescale task) to learn joint embeddings with class descriptions [32,1,3,5,23,8,9,7]. These higher-level features collapse many underlying concepts in the pursuit of class discrimination; consequentially, accessing lower-level concepts and recombining them in new ways to represent novel classes is difficult with these features.…”