A dataset for Movie Description

Abstract

Descriptive video service (DVS) provides linguistic descriptions of movies and allows visually impaired people to follow a movie along with their peers. Such descriptions are by design mainly visual and thus naturally form an interesting data source for computer vision and computational linguistics. In this work we propose a novel dataset which contains transcribed DVS, which is temporally aligned to full length HD movies. In addition we also collected the aligned movie scripts which have been used in prior work and compare the two different sources of descriptions. In total the Movie Description dataset contains a parallel corpus of over 54,000 sentences and video snippets from 72 HD movies. We characterize the dataset by benchmarking different approaches for generating video descriptions. Comparing DVS to scripts, we find that DVS is far more visual and describes precisely what is shown rather than what should happen according to the scripts created prior to movie production.

Investigation and Review of Embedded Events in Public Video Datasets

“…Several datasets [3]- [10] for action recognition are designed and opened to the public but some of them are not suitable for realistic events.…”

Towards Reliable Automatic Multimodal Content Analysis

“…Automatic Multimodal Content Analysis (AMCA), on the other hand, consists of computer-driven detection of visual and auditory elements from multimedia (Rohrbach & al 2015;Viitaniemi & al 2015). AMCA is cost-effective and produces consistent output, but is still insufficient for high-level semantic analysis.…”

Movie Description

“…6 we present and discuss the results of the LSMDC 2015 andLSMDC 2016. This work is partially based on the original publications from Rohrbach et al (2015c, b) and the technical report from Torabi et al (2015). Torabi et al (2015) collected M-VAD, Rohrbach et al (2015c) collected the MPII-MD dataset and presented the translation-based description approach. Rohrbach et al (2015b) proposed the VisualLabels approach.…”

“…(c) Focusing on more "visual" labels helps: we reduce the LSTM input dimensionality to 263 while improving the performance. (Rohrbach et al 2014), and showed the comparable performance to manually annotated SRs, see Rohrbach et al (2015c). In the following we use the best performing "Visual Labels" approach, Table 8, line (8).…”

“…The Large Scale Movie Description Challenge (LSMDC) is based on two datasets which were originally collected independently. The MPII Movie Description Dataset (MPII-MD), initially presented by Rohrbach et al (2015c), was collected from Blu-ray movie data. It consists of AD and script data and uses sentence-level manual alignment of transcribed audio to the actions in the video (Sect.…”