Common Subspace for Model and Similarity: Phrase Learning for Caption Generation from Images

Ushiku, Yoshitaka; Yamaguchi, Masataka; Mukuta, Yusuke; Harada, Tatsuya

doi:10.1109/iccv.2015.306

Cited by 56 publications

(26 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…A constrained language model is then used to generate from this representation. A conceptually related approach is pursued by Ushiku et al (2015): the authors use a common subspace model which maps all feature vectors associated with the same phrase into nearby regions of the space. For generation, a beamsearch based decoder or templates are used.…”

Section: Description As a Retrieval In Multimodal Spacementioning

confidence: 99%

Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures

Bernardi¹,

Çakıcı²,

Elliott³

et al. 2016

jair

297

236

View full text Add to dashboard Cite

Automatic description generation from natural images is a challenging problem that has recently received a large amount of interest from the computer vision and natural language processing communities. In this survey, we classify the existing approaches based on how they conceptualize this problem, viz., models that cast description as either generation problem or as a retrieval problem over a visual or multimodal representational space. We provide a detailed review of existing models, highlighting their advantages and disadvantages. Moreover, we give an overview of the benchmark image datasets and the evaluation measures that have been developed to assess the quality of machine-generated image descriptions. Finally we extrapolate future directions in the area of automatic image description generation.

show abstract

Section: Description As a Retrieval In Multimodal Spacementioning

confidence: 99%

Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures

Bernardi¹,

Çakıcı²,

Elliott³

et al. 2016

jair

297

236

View full text Add to dashboard Cite

show abstract

“…The task of generating semantic descriptions based on images can be collectively referred to as image caption. Early work [13,30,42] to generate captions is based on templates. A group of visual concepts are detected in the image and then expressed by templates.…”

Section: Scene Understandingmentioning

confidence: 99%

Learning to Understand Traffic Signs

Guo

Feng

Yin

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

One of the intelligent transportation system's critical tasks is to understand traffic signs and convey traffic information to humans. However, most related works are focused on the detection and recognition of traffic sign texts or symbols, which is not sufficient for understanding. Besides, there has been no public dataset for traffic sign understanding research. Our work takes the first step towards addressing this problem. First, we propose a "CASIA-Tencent Chinese Traffic Sign Understanding Dataset" (CTSU Dataset), which contains 5000 images of traffic signs with rich semantic descriptions. Second, we introduce a novel multi-task learning architecture that extracts text and symbol information from traffic signs, reasons the relationship between texts and symbols, classifies signs into different categories, and finally, composes the descriptions of the signs. Experiments show that the task of traffic sign understanding is achievable, and our architecture demonstrates state-of-the-art and superior performance. The CTSU Dataset is available at http://www. nlpr.ia.ac.cn/databases/CASIA-Tencent%20CTSU/index.html. CCS CONCEPTS• Computing methodologies → Scene understanding; Image representations; Spatial and physical reasoning.

show abstract

“…Similarly, in [15], the Conditional Random Field (CRF) model is implemented with the correspondence attributes, objects, and prepositions prediction to construct the sentence according to predefined templates. A new subspace embedded approach is suggested for image caption generation, called Common Subspace for Model and Similarity (CoSMoS) in [16].…”

Section: Traditional Image Captioning Modelsmentioning

confidence: 99%

Two-Tier LSTM Model for Image Caption Generation

Khaing¹,

Yu²

2021

IJIES

View full text Add to dashboard Cite

Image captioning is systematically generating the caption of the image with a sentence description. In the past few years, the automatic process of creating image caption has fascinated the great interest in Artificial Intelligence (AI) field. Image captioning defines as the basic process of building the conjunction of image processing and natural language processing at input and output position. All image processing tasks, such as the segmentation of image, object tracking, object detection, image recognition, and many others, are mostly performed using Convolutional Neural Networks (CNNs). To perform the natural language processing tasks, just as semantic role labelling, neural machine translation, speech recognition, question and answering, and many others, Recurrent Neural Networks (RNNs) and long-term memory networks (LSTMs) are essential for some of the biggest breakthroughs. This paper proposes the efficient encoder-decoder framework-based image captioning model, namely Two-Tier LSTM (TT-LSTM) Model. The TT-LSTM model is designedly implemented upon the encoder-decoder framework with two LSTM layers. This research is implemented on the MSCOCO, Flickr30k, and Flickr8k datasets; and evaluated with standard evaluation matrixes such as ROUGE-L, CIDEr, and four BLEU scores. The outcomes of the experiments on the typical datasets reveal that the proposed model generates meaningful natural language sentences. The proposed model also improves the sentence generation efficiency and can achieve better performance for image caption generation.

show abstract

Common Subspace for Model and Similarity: Phrase Learning for Caption Generation from Images

Cited by 56 publications

References 25 publications

Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures

Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures

Learning to Understand Traffic Signs

Two-Tier LSTM Model for Image Caption Generation

Contact Info

Product

Resources

About