Video description: A comprehensive survey of deep learning approaches

Abstract

Video description refers to understanding visual content and transforming that acquired understanding into automatic textual narration. It bridges the key AI fields of computer vision and natural language processing in conjunction with real-time and practical applications. Deep learning-based approaches employed for video description have demonstrated enhanced results compared to conventional approaches. The current literature lacks a thorough interpretation of the recently developed and employed sequence to sequence techniques for video description. This paper fills that gap by focusing mainly on deep learning-enabled approaches to automatic caption generation. Sequence to sequence models follow an Encoder–Decoder architecture employing a specific composition of CNN, RNN, or the variants LSTM or GRU as an encoder and decoder block. This standard-architecture can be fused with an attention mechanism to focus on a specific distinctiveness, achieving high quality results. Reinforcement learning employed within the Encoder–Decoder structure can progressively deliver state-of-the-art captions by following exploration and exploitation strategies. The transformer mechanism is a modern and efficient transductive architecture for robust output. Free from recurrence, and solely based on self-attention, it allows parallelization along with training on a massive amount of data. It can fully utilize the available GPUs for most NLP tasks. Recently, with the emergence of several versions of transformers, long term dependency handling is not an issue anymore for researchers engaged in video processing for summarization and description, or for autonomous-vehicle, surveillance, and instructional purposes. They can get auspicious directions from this research.

Local feature‐based video captioning with multiple classifier and CARU‐attention

“…This involved creating coherent and specific sentences by selecting keywords based on predefined templates. However, this approach had limitations in supporting complex event representation and understanding multiple scenes in long videos because it is restricted to simple sentence structures [12,13]. Video captioning approaches can be broadly divided into two types: template-based approaches and sequence-based approaches.…”

“…In order to present our results, we have performed a clear comparison of our proposed model with the selected methods available for the video captioning task. These methods include EEDVC [52], DCE [53], MFT [54], WLT [55], MDVC [12], EMVC [56], BMT [57], PPVC [38], and PDVC [35]. The performance analysis and evaluation of these methods are reported in Table 1.…”

An innovative deep active learning approach for improving unlabeled audio classification by selectively querying informative instance

“…This iterative process continues until the stopping criterion is met. 15 L: the set of labeled instances U: the set of unlabeled instances φ (u¡): query strategy where u¡ E u B: number of instances to be selected at each iteration (batch size)…”

Parallel Dense Video Caption Generation with Multi-Modal Features

“…Dense video captioning involves visual understanding processes that locate different events in a video and generate descriptive captions for each interesting object. This approach represents video content in detail by transforming frame sequences into multiple descriptive sentences among multiple clips in a long video [26][27][28]. In the research of Shen et al [29], dense image captioning is migrated to the video field by combining the multi-scale suggestion module and a visual context perception mechanism.…”

“…This work contrasted our proposed model with the state-of-the-art methods for the dense video captioning task, consisting of EEDVC [43], DCE [8], MFT [26], WLT [27], SDVC [9], EHVC [31], MDVC [28], BMT [52], EMVC [51], PPVC [13], and PDVC [46]. The contrast results are displayed in Table 1.…”