Learning Models for Object Recognition from Natural Language Descriptions

Abstract

We investigate the task of learning models for visual object recognition from natural language descriptions alone. The approach contributes to the recognition of fine-grain object categories, such as animal and plant species, where it may be difficult to collect many images for training, but where textual descriptions of visual attributes are readily available. As an example we tackle recognition of butterfly species, learning models from descriptions in an online nature guide. We propose natural language processing methods for extracting salient visual attributes from these descriptions to use as 'templates' for the object categories, and apply vision methods to extract corresponding attributes from test images. A generative model is used to connect textual terms in the learnt templates to visual attributes. We report experiments comparing the performance of humans and the proposed method on a dataset of ten butterfly categories.

Robust gender recognition by exploiting facial attributes dependencies

“…Attribute-based representations have recently received much attention because they have been successfully used for image retrieval (Yu et al, 2012), for recognizing objects (Duan et al, 2012;Wang et al, 2009), for describing unknown objects (Farhadi et al, 2009), and even for learning new unseen object models from descriptions (Farhadi et al, 2009;Lampert et al, 2009). Facial attributes have a key role in human-computer interaction applications, image and video retrieval and surveillance.…”

Harvesting Training Images for Fine-Grained Object Categories Using Visual Descriptions

“…We illustrate harvesting training images for ten butterfly categories of the Leeds Butterfly Dataset [15], using the provided eNature visual descriptions. Figure 2 shows the pipeline for our method, starting from the butterfly species' name and visual description.…”

“…The seed phrases are restricted to noun phrases and adjective phrases, obtained via phrase chunking as in [15]. The number of seed phrases per category ranges from 5 to 17 depending on the length of the description; an example list is shown in Fig.…”

Learning beautiful attributes

“…This is a cost-effective alternative to hand-listing attributes [10,15] and to architectures which require a human-in-the-loop [25]. Existing solutions [1,34,35] were typically developed for visual object recognition tasks. [34] proposes to mine pre-existing natural language resources.…”

“…Existing solutions [1,34,35] were typically developed for visual object recognition tasks. [34] proposes to mine pre-existing natural language resources. [1] uses mutual information to learn attributes relevant for e-commerce categories (handbags, shoes, earrings and ties) [8] uses latent CRF to discover detectable and discriminative attributes.…”