Unsupervised Learning from Narrated Instruction Videos

Abstract

We address the problem of automatically learning the main steps to complete a certain task, such as changing a car tire, from a set of narrated instruction videos. The contributions of this paper are three-fold. First, we develop a new unsupervised learning approach that takes advantage of the complementary nature of the input video and the associated narration. The method solves two clustering problems, one in text and one in video, applied one after each other and linked by joint constraints to obtain a single coherent sequence of steps in both modalities. Second, we collect and annotate a new challenging dataset of real-world instruction videos from the Internet. The dataset contains about 800,000 frames for five different tasks 1 that include complex interactions between people and objects, and are captured in a variety of indoor and outdoor settings. Third, we experimentally demonstrate that the proposed method can automatically discover, in an unsupervised manner, the main steps to achieve the task and locate the steps in the input videos.

Neural Image Caption Generation with Weighted Training and Reference

“…Please note that there are other captioning tasks that are related to our research, such as dense captioning [22] and video captioning [1,48].…”

“…We propose a weighting strategy based on words' similarity by considering the semantic information. For word s i , suppose the synonym set ('synset') of its kth meaning is ss ik by WordNet, 1 we compute the weight of the synset as follows:…”

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

“…Among them, instructional videos provide more intuitive visual examples, and will be focused on in this paper. With the explosion of video data on the Internet, people around the world have uploaded and watched substantial instructional videos [6], [59], covering miscellaneous categories. As suggested by the scientists in educational psychology [54], novices often face difficulties in learning from the whole realistic task, and it is necessary to divide the whole task into smaller segments or steps as a form of simplification.…”

“…Accordingly, a variety of relative tasks have been studied by morden computer vision community in recent years (e.g., action temporal localization [74], [80], video summarization [23], [49], [79] and video caption [35], [77], [83], etc). Also, increasing efforts have been devoted to exploring different challenges of instructional video analysis [6], [31], [59], [82] evidence, Fig. 2 shows the growing number of publications in the top venues over the recent ten years.…”

“…In the meantime, a number of datasets for instructional video analysis [6], [15], [36], [56], [65], [70], [82] have been collected in the community. Annotated with texts and temporal boundaries of a series of steps to complete different tasks, these datasets have provided good benchmarks for preliminary research.…”