Long-term Recurrent Convolutional Networks for Visual Recognition and Description

Donahue, Jeff; Hendricks, Lisa Anne; Rohrbach, Marcus; Venugopalan, Subhashini; Guadarrama, Sergio; Saenko, Kate; Darrell, Trevor

doi:10.48550/arxiv.1411.4389

Cited by 65 publications

(121 citation statements)

References 0 publications

Supporting

Mentioning

120

Contrasting

Order By: Relevance

“…1. The accuracy gap between ours and those reported in [7,25,1,20,26] is mainly caused by the availability of different input frames at test time.…”

Section: Experimental Settingcontrasting

confidence: 84%

“…There are many successful video recognition models. For instance, [7] proposed a class of recurrent long-term models that can be jointly trained to learn temporal dynamics and convolutional perceptual representations, and demonstrated superior performance on recognition and description of images and videos. [25] addressed the problem of learning spatio tempora features for videos using 3D ConvNets.…”

Section: Related Workmentioning

confidence: 99%

“…We evaluate our approach on two popular video classification datasets, namely, UCF101 [22] and HMDB51 [12]. We select six victim models from a wide range of video classification methods, including CNN+LSTM [7], C3D [25], ResNet3D [9], P3D [20], and I3D [1] on both ResNet [10] and Inception [23]. We start with the white-box attacks, in which the parameters of the victim models are known to the attacker, and then transfer the perturbations computed on each video to other unseen models, i.e., the black-box attacks.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Appending Adversarial Frames for Universal Video Attack

Chen

Xie

Pang

et al. 2019

Preprint

View full text Add to dashboard Cite

There have been many efforts in attacking image classification models with adversarial perturbations, but the same topic on video classification has not yet been thoroughly studied. This paper presents a novel idea of videobased attack, which appends a few dummy frames (e.g., containing the texts of 'thanks for watching') to a video clip and then adds adversarial perturbations only on these new frames. Our approach enjoys three major benefits, namely, a high success rate, a low perceptibility, and a strong ability in transferring across different networks. These benefits mostly come from the common dummy frame which pushes all samples towards the boundary of classification. On the other hand, such attacks are easily to be concealed since most people would not notice the abnormality behind the perturbed video clips. We perform experiments on two popular datasets with six state-of-the-art video classification models, and demonstrate the effectiveness of our approach in the scenario of universal video attacks.

show abstract

“…1. The accuracy gap between ours and those reported in [7,25,1,20,26] is mainly caused by the availability of different input frames at test time.…”

Section: Experimental Settingcontrasting

confidence: 84%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Appending Adversarial Frames for Universal Video Attack

Chen

Xie

Pang

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…We explore three state-of-the-art action recognition methods: Long-term Recurrent Convolutional Networks (LRCN) [2], 3D Convolutional Networks (C3D) [12] and Temporal Shift Module (TSM) [7].…”

Section: Action Recognition Methodsmentioning

confidence: 99%

Vision-based Behavioral Recognition of Novelty Preference in Pigs

Shirke,

Golden,

Gautam

et al. 2021

Preprint

View full text Add to dashboard Cite

Behavioral scoring of research data is crucial for extracting domain-specific metrics but is bottlenecked on the ability to analyze enormous volumes of information using human labor. Deep learning is widely viewed as a key advancement to relieve this bottleneck. We identify one such domain, where deep learning can be leveraged to alleviate the process of manual scoring. Novelty preference paradigms have been widely used to study recognition memory in pigs, but analysis of these videos requires human intervention. We introduce a subset of such videos in the form of the 'Pig Novelty Preference Behavior' (PNPB) dataset that is fully annotated with pig actions and keypoints. In order to demonstrate the application of state-of-the-art action recognition models on this dataset, we compare LRCN, C3D, and TSM on the basis of various analytical metrics and discuss common pitfalls of the models. Our methods achieve an accuracy of 93% and a mean Average Precision of 96% in estimating piglet behavior. We open-source our code and annotated dataset at https://github.com/ AIFARMS/NOR-behavior-recognition

show abstract

“…Our "self talk" framework has two "executives" that takes their roles iteratively: 1) question generation, which is responsible for asking the right questions, and 2) question answering, which accepts the questions and generate potential answers. With the rapid development in computer vision and machine learning [Mao et al, 2014;Donahue et al, 2014;Karpathy and Li, 2014;Vinyals et al, 2014; Figure 1: One example self talk by the presented system, while the affirmative or questionable answer is decided by confidence score from visual answering executive. Chen and Zitnick, 2014] there are a few tools developed for this seemingly intuitive philosophy in Artificial Intelligence, but self-talk is certainly beyond the aggregation of tools, because it is fundamentally a challenging chicken egg problem.…”

Section: Introductionmentioning

confidence: 99%

Neural Self Talk: Image Understanding via Continuous Questioning and Answering

Yang¹,

Li²,

Fermüller³

et al. 2015

Preprint

View full text Add to dashboard Cite

In this paper we consider the problem of continuously discovering image contents by actively asking image based questions and subsequently answering the questions being asked. The key components include a Visual Question Generation (VQG) module and a Visual Question Answering module, in which Recurrent Neural Networks (RNN) and Convolutional Neural Network (CNN) are used. Given a dataset that contains images, questions and their answers, both modules are trained at the same time, with the difference being VQG uses the images as input and the corresponding questions as output, while VQA uses images and questions as input and the corresponding answers as output. We evaluate the self talk process subjectively using Amazon Mechanical Turk, which show effectiveness of the proposed method.

show abstract

Long-term Recurrent Convolutional Networks for Visual Recognition and Description

Cited by 65 publications

References 0 publications

Appending Adversarial Frames for Universal Video Attack

Appending Adversarial Frames for Universal Video Attack

Vision-based Behavioral Recognition of Novelty Preference in Pigs

Neural Self Talk: Image Understanding via Continuous Questioning and Answering

Contact Info

Product

Resources

About