Words Matter: Scene Text for Image Classification and Retrieval

Karaoğlu, Sezer; Tao, Ran; Gevers, Theo; Smeulders, A.W.M.

doi:10.1109/tmm.2016.2638622

Cited by 110 publications

(69 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The most related work with our proposal in this paper is the one in [3], [20], where scene texts in the business place images are considered as the domain knowledge. The authors proposed to combine visual and textual cues for fine-grained business place classification.…”

Section: Related Work a Fine-grained Classificationmentioning

confidence: 99%

“…In [20], Bagof-Words are used to represent visual cues. Deep visual cues given by GoogleNet [5] are used in [3]. Sharing the same spirit as part-based methods, the authors proposed in [3] to extract 100 EdgeBoxes [21].…”

Section: Related Work a Fine-grained Classificationmentioning

confidence: 99%

“…Following [3], [20], we also propose to combine visual and text cues to improve fine-grained classification. We combine the visual cues and textual cues at the feature level.…”

Section: Multi-modal Fusionmentioning

confidence: 99%

“…For instance, texts on the buildings or wrapping bags are quite useful for distinguishing among their categories. With the recent developments in text detection and recognition in the wild [1], [2], it has been shown that textual cues are very beneficial to fine-grained classification, especially in the classification of business places such as bakery, cafe and bookstore [3].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Integrating Scene Text and Visual Appearance for Fine-Grained Image Classification

et al. 2018

View full text Add to dashboard Cite

Text in natural images contains rich semantics that are often highly relevant to objects or scene. In this paper, we focus on the problem of fully exploiting scene text for visual understanding. The main idea is combining word representations and deep visual features into a globally trainable deep convolutional neural network. First, the recognized words are obtained by a scene text reading system. Then, we combine the word embedding of the recognized words and the deep visual features into a single representation, which is optimized by a convolutional neural network for fine-grained image classification. In our framework, the attention mechanism is adopted to reveal the relevance between each recognized word and the given image, which further enhances the recognition performance. We have performed experiments on two datasets: Con-Text dataset and Drink Bottle dataset, that are proposed for fine-grained classification of business places and drink bottles, respectively. The experimental results consistently demonstrate that the proposed method combining textual and visual cues significantly outperforms classification with only visual representations. Moreover, we have shown that the learned representation improves the retrieval performance on the drink bottle images by a large margin, making it potentially useful in product search.

show abstract

Section: Related Work a Fine-grained Classificationmentioning

confidence: 99%

Section: Related Work a Fine-grained Classificationmentioning

confidence: 99%

“…Following [3], [20], we also propose to combine visual and text cues to improve fine-grained classification. We combine the visual cues and textual cues at the feature level.…”

Section: Multi-modal Fusionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Integrating Scene Text and Visual Appearance for Fine-Grained Image Classification

et al. 2018

View full text Add to dashboard Cite

show abstract

“…Scene text retrieval [11,9], which aims at retrieving images based on text content, is closely related to scene text verification. The verification task could be seen as a subtask for scene text retrieval, as it only cares about the existence of text and no ranking is needed.…”

Section: Related Workmentioning

confidence: 99%

Large Scale Scene Text Verification with Guided Attention

Gorban

et al. 2019

Computer Vision – ACCV 2018

View full text Add to dashboard Cite

Many tasks are related to determining if a particular text string exists in an image. In this work, we propose a new framework that learns this task in an end-to-end way. The framework takes an image and a text string as input and then outputs the probability of the text string being present in the image. This is the first end-to-end framework that learns such relationships between text and images in scene text area. The framework does not require explicit scene text detection or recognition and thus no bounding box annotations are needed. It is also the first work in scene text area that tackles such a weakly labeled problem. Based on this framework, we developed a model called Guided Attention. Our designed model achieves better results than several state-of-the-art scene text reading based solutions for a challenging Street View Business Matching task. The task tries to find correct business names for storefront images and the dataset we collected for it is substantially larger, and more challenging than existing scene text dataset. This new real-world task provides a new perspective for studying scene text related problems.

show abstract

ST-Sem: A Multimodal Method for Points-of-Interest Classification Using Street-Level Imagery

Noorian¹,

Psyllidis²,

Bozzon³

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Street-level imagery contains a variety of visual information about the facades of Points of Interest (POIs). In addition to general morphological features, signs on the facades of, primarily, business-related POIs could be a valuable source of information about the type and identity of a POI. Recent advancements in computer vision could leverage visual information from street-level imagery, and contribute to the classification of POIs. However, there is currently a gap in existing literature regarding the use of visual labels contained in street-level imagery, where their value as indicators of POI categories is assessed. This paper presents Scene-Text Semantics (ST-Sem), a novel method that leverages visual labels (e.g., texts, logos) from street-level imagery as complementary information for the categorization of business-related POIs. Contrary to existing methods that fuse visual and textual information at a featurelevel, we propose a late fusion approach that combines visual and textual cues after resolving issues of incorrect digitization and semantic ambiguity of the retrieved textual components. Experiments on two existing and a newly-created datasets show that ST-Sem can outperform visual-only approaches by 80% and related multimodal approaches by 4%.

show abstract

Words Matter: Scene Text for Image Classification and Retrieval

Cited by 110 publications

References 57 publications

Integrating Scene Text and Visual Appearance for Fine-Grained Image Classification

Integrating Scene Text and Visual Appearance for Fine-Grained Image Classification

Large Scale Scene Text Verification with Guided Attention

ST-Sem: A Multimodal Method for Points-of-Interest Classification Using Street-Level Imagery

Contact Info

Product

Resources

About