Automatic Image and Video Caption Generation With Deep Learning: A Concise Review and Algorithmic Overlap

Amirian, Soheyla; Rasheed, Khaled; Taha, Thiab R.; Arabnia, Hamid R.

doi:10.1109/access.2020.3042484

Cited by 72 publications

(24 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is currently done using deep models that combine image embedding with the ability to generate descriptions using recurrent Long short-term memory (LSTM) neural layers such as [ 41 , 42 , 43 ]. A large survey on this field can be found at [ 44 , 45 ]. Thus, one could use such automatically generated image descriptions with sentences that have been generated by the text summarizer (see Section 2.1.1 ) to determine the similarity between the text and the various images we have, in order to select the image that minimises the given distance metric [ 46 ].…”

Section: Materials and Methodsmentioning

confidence: 99%

TIPS: A Framework for Text Summarising with Illustrative Pictures

Golec

Hachaj

Sokal

2021

Entropy

View full text Add to dashboard Cite

We propose an algorithm to generate graphical summarising of longer text passages using a set of illustrative pictures (TIPS). TIPS is an algorithm using a voting process that uses results of individual “weak” algorithms. The proposed method includes a summarising algorithm that generates a digest of the input document. Each sentence of the text summary is used as the input for further processing by the sentence transformer separately. A sentence transformer performs text embedding and a group of CLIP similarity-based algorithms trained on different image embedding finds semantic distances between images in the illustration image database and the input text. A voting process extracts the most matching images to the text. The TIPS algorithm allows the integration of the best (highest scored) results of the different recommendation algorithms by diminishing the influence of images that are a disjointed part of the recommendations of the component algorithms. TIPS returns a set of illustrative images that describe each sentence of the text summary. Three human judges found that the use of TIPS resulted in an increase in matching highly relevant images to text, ranging from 5% to 8% and images relevant to text ranging from 3% to 7% compared to the approach based on single-embedding schema.

show abstract

Section: Materials and Methodsmentioning

confidence: 99%

TIPS: A Framework for Text Summarising with Illustrative Pictures

Golec

Hachaj

Sokal

2021

Entropy

View full text Add to dashboard Cite

show abstract

“…Proposition (1) in fact shows that in visual tracking the correlation filter and the convolution filter are equivalent in the sense of equal minimum mean-square error of estimation, under the condition that the ideal filter response is a 2-D centrosymmetric Gaussian function and the optimal solutions exist. More specifically, * As long as the optimal solution of one of the filters is known, the optimal solution of the other can be obtained immediately; * The mean-square errors of the filter responses of the two optimal solutions to a given detection sample are equal; * The filter responses of the two optimal solutions to a given detection sample are symmetric about the origin.…”

Section: Proof Of the Equivalencementioning

confidence: 99%

“…Two currently predominant approaches are discriminative correlation filter (DCF)-based methods and deep learning (DL)-based methods. Deep learning has been intensively studied and demonstrated remarkable success in a wide range of computer vision areas, such as image classification, object detection, image caption and semantic segmentation, etc [23,35,16,48,1]. Inspired by deep learning breakthroughs in these fields, DL-based methods have attracted considerable interest in the visual tracking community and witness rapid development and great advances in recent years.…”

Section: Introductionmentioning

confidence: 99%

Equivalence of Correlation Filter and Convolution Filter in Visual Tracking

Li¹,

Zhao²,

Feng³

et al. 2021

Preprint

View full text Add to dashboard Cite

Discriminative) Correlation Filter has been successfully applied to visual tracking and has advanced the field significantly in recent years. Correlation filter-based trackers consider visual tracking as a problem of matching the feature template of the object and candidate regions in the detection sample, in which correlation filter provides the means to calculate the similarities. In contrast, convolution filter is usually used for blurring, sharpening, embossing, edge detection, etc in image processing. On the surface, correlation filter and convolution filter are usually used for different purposes. In this paper, however, we proves, for the first time, that correlation filter and convolution filter are equivalent in the sense that their minimum mean-square errors (MMSEs) in visual tracking are equal, under the condition that the optimal solutions exist and the ideal filter response is Gaussian and centrosymmetric. This result gives researchers the freedom to choose correlation or convolution in formulating their trackers. It also suggests that the explanation of the ideal response in terms of similarities is not essential.

show abstract

“…Many researchers present different models on video captioning [10,25,24,17,27], mostly with limited success and many constraints. In video captioning [4], Krishna et al [17], however, presented Densecaptioning, which focuses on detecting multiple events that occur in a video by jointly localizing temporal proposals of interest and then describing each with natural language. This model introduced a new captioning module that uses contextual information from past and future events to describe all events jointly.…”

Section: Image and Video Captioningmentioning

confidence: 99%

An Integrated Approach for Video Captioning and Applications

Amirian¹,

Taha²,

Rasheed³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Physical computing infrastructure, data gathering, and algorithms have recently had significant advances to extract information from images and videos. The growth has been especially outstanding in image captioning and video captioning. However, most of the advancements in video captioning still take place in short videos. In this research, we caption longer videos only by using the keyframes, which are a small subset of the total video frames. Instead of processing thousands of frames, only a few frames are processed depending on the number of keyframes. There is a trade-off between the computation of many frames and the speed of the captioning process. The approach in this research is to allow the user to specify the trade-off between execution time and accuracy. In addition, we argue that linking images, videos, and natural language offers many practical benefits and immediate practical applications. From the modeling perspective, instead of designing and staging explicit algorithms to process videos and generate captions in complex processing pipelines, our contribution lies in designing hybrid deep learning architectures to apply in long videos by captioning video keyframes. We consider the technology and the methodology that we have developed as steps toward the applications discussed in this research.

show abstract

Automatic Image and Video Caption Generation With Deep Learning: A Concise Review and Algorithmic Overlap

Cited by 72 publications

References 57 publications

TIPS: A Framework for Text Summarising with Illustrative Pictures

TIPS: A Framework for Text Summarising with Illustrative Pictures

Equivalence of Correlation Filter and Convolution Filter in Visual Tracking

An Integrated Approach for Video Captioning and Applications

Contact Info

Product

Resources

About