The Traffic Scene Understanding and Prediction Based on Image Captioning

Li, Wei; Qu, Zhaowei; Song, Haiyu; Wang, Pengjie; Xue, Bo

doi:10.1109/access.2020.3047091

Cited by 23 publications

(7 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Li et al [13] have demonstrated that image captioning of traffic scenes can provide richer semantic information for Advanced Driver Assistance Systems (ADAS) to make decisions. Appropriate driving suggestions generated from captions can improve driver safety.…”

Section: Research Related To Image Captioning In Traffic Scenesmentioning

confidence: 99%

TSIC-CLIP: Traffic Scene Image Captioning Model Based on Clip

Zhang,

Xu,

et al. 2024

ITC

View full text Add to dashboard Cite

Image captioning in traffic scenes presents several challenges, including imprecise caption generation, lack of personalization, and an unwieldy number of model parameters. We propose a new image captioning model for traffic scenes to address these issues. The model incorporates an adapter-based fine-tuned feature extraction part to enhance personalization and a caption generation module using global weighted attention pooling to reduce model parameters and improve accuracy. The proposed model consists of four main stages. In the first stage, the Image-Encoder extracts the global features of the input image and divides it into nine sub-regions, encoding each sub-region separately. In the second stage, the Text-Encoder encodes the text dataset to obtain text features. It then calculates the similarity between the image sub-region features and encoded text features, selecting the text features with the highest similarity. Subsequently, the pre-trained Faster RCNN model extracts local image features. The model then splices together the text features, global image features, and local image features to fuse the multimodal information. In the final stage, the extracted features are fed into the Captioning model, which effectively fuses the different features using a novel global weighted attention pooling layer. The Captioning model then generates natural language image captions. The proposed model is evaluated on the MS-COCO dataset, Flickr 30K dataset, and BUUISE-Image dataset, using mainstream evaluation metrics. Experiments demonstrate significant improvements across all evaluation metrics on the public datasets and strong performance on the BUUISE-Image traffic scene dataset.

show abstract

Section: Research Related To Image Captioning In Traffic Scenesmentioning

confidence: 99%

TSIC-CLIP: Traffic Scene Image Captioning Model Based on Clip

Zhang,

Xu,

et al. 2024

ITC

View full text Add to dashboard Cite

show abstract

“…The BIC frameworks justify the semantic information of the image; SMIC frameworks identify the people, places, and locations uploaded on social media websites such as Facebook, Instagram, Twitter, etc., and create their captions. The VAC can further be categorized into Traffic Image Captioning (TIC) [12], which benefits drivers on crowded streets and highways. Furthermore, TIC frameworks are also utilized in autonomous vehicles.…”

Section: Generic Purpose Image Captioning (Gpic)mentioning

confidence: 99%

Domain-specific image captioning: a comprehensive review

Sharma,

Padha

2024

Int J Multimed Info Retr

View full text Add to dashboard Cite

Image captioning is a sentence summarizing the semantic content of an image. Describing the essential information of an image in the form of a statement typically deals with computer vision and natural language processing. The prior research frequently addressed this subject simply using standard machine learning approaches that model such systems by extracting the hand-engineered characteristics from the input data. With the resurgence of deep-learning approaches, the image captioning area has experienced state-of-the-art outcomes in numerous application domains, including visual aid, traffic assistance, medical support, and remote sensing. This research thoroughly reviews the application domain-based image captioning frameworks by addressing both classical and modern deep learning-based architectures. We further propose a taxonomy to characterize the same by studying their distinct technological foundations and performances across diverse datasets. We also describe evaluation metrics used to examine the captioning image models across several domains. Finally, we highlight open research problems concerning such models from a futuristic standpoint.

show abstract

“…To ensure sustainable transportation, big data analysis through methods such as [1] and data support through IoT data security transmission technologies such as [2] are used to provide decisions for transportation planning. Deep learning also has an indelible role in this, and natural language description of traffic scenes is important for assisting visually impaired people in their daily lives and in participating in traffic [3,4], as well as generating rich semantic information for drivers, thus assisting in the generation of intelligent decision suggestions, reducing driver decision time, and being important for reducing the risk of accidents [5]. This maintains the resilience of traffic as well as the sustainability of traffic by ensuring road safety.…”

Section: Introductionmentioning

confidence: 99%

Traffic Scene Captioning with Multi-Stage Feature Enhancement

Zhang,

Ma,

Liu

et al. 2023

Computers, Materials &Amp; Continua

View full text Add to dashboard Cite

Traffic scene captioning technology automatically generates one or more sentences to describe the content of traffic scenes by analyzing the content of the input traffic scene images, ensuring road safety while providing an important decision-making function for sustainable transportation. In order to provide a comprehensive and reasonable description of complex traffic scenes, a traffic scene semantic captioning model with multi-stage feature enhancement is proposed in this paper. In general, the model follows an encoder-decoder structure. First, multilevel granularity visual features are used for feature enhancement during the encoding process, which enables the model to learn more detailed content in the traffic scene image. Second, the scene knowledge graph is applied to the decoding process, and the semantic features provided by the scene knowledge graph are used to enhance the features learned by the decoder again, so that the model can learn the attributes of objects in the traffic scene and the relationships between objects to generate more reasonable captions. This paper reports extensive experiments on the challenging MS-COCO dataset, evaluated by five standard automatic evaluation metrics, and the results show that the proposed model has improved significantly in all metrics compared with the state-of-the-art methods, especially achieving a score of 129.0 on the CIDEr-D evaluation metric, which also indicates that the proposed model can effectively provide a more reasonable and comprehensive description of the traffic scene.

show abstract

The Traffic Scene Understanding and Prediction Based on Image Captioning

Cited by 23 publications

References 57 publications

TSIC-CLIP: Traffic Scene Image Captioning Model Based on Clip

TSIC-CLIP: Traffic Scene Image Captioning Model Based on Clip

Domain-specific image captioning: a comprehensive review

Traffic Scene Captioning with Multi-Stage Feature Enhancement

Contact Info

Product

Resources

About