Long-term Recurrent Convolutional Networks for Visual Recognition and Description

Abstract

Abstract-Models based on deep convolutional networks have dominated recent image interpretation tasks; we investigate whether models which are also recurrent are effective for tasks involving sequences, visual and otherwise. We describe a class of recurrent convolutional architectures which is end-to-end trainable and suitable for large-scale visual understanding tasks, and demonstrate the value of these models for activity recognition, image captioning, and video description. In contrast to previous models which assume a fixed visual representation or perform simple temporal averaging for sequential processing, recurrent convolutional models are "doubly deep" in that they learn compositional representations in space and time. Learning long-term dependencies is possible when nonlinearities are incorporated into the network state updates. Differentiable recurrent models are appealing in that they can directly map variable-length inputs (e.g., videos) to variable-length outputs (e.g., natural language text) and can model complex temporal dynamics; yet they can be optimized with backpropagation. Our recurrent sequence models are directly connected to modern visual convolutional network models and can be jointly trained to learn temporal dynamics and convolutional perceptual representations. Our results show that such models have distinct advantages over state-of-the-art models for recognition or generation which are separately defined or optimized.

Learning to Track at 100 FPS with Deep Regression Networks

“…In many other areas of computer vision, such as image classification, object detection, segmentation, or activity recognition, machine learning has allowed vision algorithms to train from offline data and learn about the world [5,23,13,25,9,28]. In each of these cases, the performance of the algorithm improves as it iterates through the training set of images.…”

Scalable Multitask Representation Learning for Scene Classification

“…Xiao et al [11] 20.9 28.1 38.0 Su and Jurie [10] 35.6 (0.4) Donahue et al [3] 40.9 (0. Table 1: Mean accuracy (%) and standard deviation across 10 splits on the SUN397 dataset.…”

Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models

“…dataset (Young et al 2014), a popular benchmark for caption generation and retrieval that has been used, among others, by Chen and Zitnick (2015); Donahue et al (2015); Fang et al (2015); Gong et al (2014b); Karpathy et al (2014); Karpathy and Fei-Fei (2015); Kiros et al (2014); Klein et al (2014); Lebret et al (2015); Mao et al (2015); Vinyals et al (2015); Xu et al (2015). Flickr30k contains 31,783 images focusing mainly on people and animals, and 158,915 English captions (five per image).…”

“…As mentioned in the Introduction, the most common imagelanguage understanding task in the literature is automatic image captioning (Chen and Zitnick 2015;Donahue et al 2015;Fang et al 2015;Farhadi et al 2010;Hodosh et al 2013;Karpathy and Fei-Fei 2015;Kiros et al 2014;Klein et al 2014;Lev et al 2016;Kulkarni et al 2011;Lebret et al 2015;Ma et al 2015;Mao et al 2015;Ordonez et al 2011;Vinyals et al 2015;Yao et al 2010). Of most importance to us are the methods attempting to associate local regions in an image with words or phrases in the captions, as they would likely benefit the most from our annotations.…”

“…There has been a recent surge of work in this area, and in particular, on the task of sentence-based image description (Chen and Zitnick 2015;Donahue et al 2015;Fang et al 2015;Farhadi et al 2010;Hodosh et al 2013;Karpathy and Fei-Fei 2015;Kiros et al 2014;Klein et al 2014;Kulkarni et al 2011;Lebret et al 2015;Mao et al 2015;Ordonez et al 2011;Vinyals et al 2015;Yao et al 2010) and visual question answering (Antol et al 2015;Gao et al 2015;Krishna et al 2016;Malinowski and Fritz 2014;Ren et al 2015;Yu et al 2015). Unfortunately, due to a lack of datasets that provide not only paired sentences and images, but detailed grounding of specific phrases in image regions, most of these methods attempt to directly learn mappings from whole images to whole sentences.…”