Context-Adaptive-Based Image Captioning by Bi-CARU

Im, Sio-Kei; Chan, Ka-Hou

doi:10.1109/access.2023.3302512

Cited by 5 publications

(2 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, taking cues from image captioning techniques can be directly applied to video captioning, allowing the exploration of correlations between frame sequences [19][20][21].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Local feature‐based video captioning with multiple classifier and CARU‐attention

Im,

Chan

2024

IET Image Processing

Self Cite

View full text Add to dashboard Cite

Video captioning aims to identify multiple objects and their behaviours in a video event and generate captions for the current scene. This task aims to generate a detailed description of the current video in real‐time using natural language, which requires deep learning to analyze and determine the relationships between interesting objects in the frame sequence. In practice, existing methods typically involve detecting objects in the frame sequence and then generating captions based on features extracted through object coverage locations. Therefore, the results of caption generation are highly dependent on the performance of object detection and identification. This work proposes an advanced video captioning approach that works in adaptively and effectively addresses the interdependence between event proposals and captions. Additionally, an attention‐based multimodel framework is introduced to capture the main context from the frame and sound in the video scene. Also, an intermediate model is presented to collect the hidden states captured from the input sequence, which performs to extract the main features and implicitly produce multiple event proposals. For caption prediction, the proposed method employs the CARU layer with attention consideration as the primary RNN layer for decoding. Experimental results showed that the proposed work achieves improvements compared to the baseline method and also better performance compared to other state‐of‐the‐art models on the ActivityNet dataset, presenting competitive results in the tasks of video captioning.

show abstract

“…In addition, taking cues from image captioning techniques can be directly applied to video captioning, allowing the exploration of correlations between frame sequences [19][20][21].…”

Section: Related Workmentioning

confidence: 99%

“…aiming to enhance convergence during the training process [21,45,46]. Respond to such modification, a trainable procedure is introduced for the weight can adaptively adjust caption information, improving the decoded states as below:…”

Section: Caru-attentionmentioning

confidence: 99%