Reduced Analytic Dependency Modeling: Robust Fusion for Visual Recognition

Andy, J.; Yuen, Pong C.

doi:10.1007/s11263-014-0723-7

Cited by 25 publications

(7 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In spite of that, to get an approximate idea about how GeThR-Net performs compared to these methods, we provide some comparisons. The mAP reported on CCV by some of the recent methods are: 70.6 [40], 64.0 [46], 63.4 [17], 60.3 [43], 68.2 [16], 64.0 [10] and 83.5 [41]. We perform better (mAP of 79.3) than six of these methods.…”

Section: Discussion On Resultsmentioning

confidence: 91%

See 1 more Smart Citation

GeThR-Net: A Generalized Temporally Hybrid Recurrent Neural Network for Multimodal Information Fusion

Gandhi¹,

Sharma²,

Biswas³

et al. 2016

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Data generated from real world events are usually temporal and contain multimodal information such as audio, visual, depth, sensor etc. which are required to be intelligently combined for classification tasks. In this paper, we propose a novel generalized deep neural network architecture where temporal streams from multiple modalities are combined. There are total M+1 (M is the number of modalities) components in the proposed network. The first component is a novel temporally hybrid Recurrent Neural Network (RNN) that exploits the complimentary nature of the multimodal temporal information by allowing the network to learn both modality specific temporal dynamics as well as the dynamics in a multimodal feature space. M additional components are added to the network which extract discriminative but non-temporal cues from each modality. Finally, the predictions from all of these components are linearly combined using a set of automatically learned weights. We perform exhaustive experiments on three different datasets spanning four modalities. The proposed network is relatively 3.5%, 5.7% and 2% better than the best performing temporal multimodal baseline for UCF-101, CCV and Multimodal Gesture datasets respectively.

show abstract

Section: Discussion On Resultsmentioning

confidence: 91%

“…The dataset has 393 training, 287 testing, and 276 testing sequences. Each sequence is of duration between 1-2 minutes and contains [8][9][10][11][12][13][14][15][16][17][18][19][20] gestures. Furthermore, the test sequences also have 'distracter' (out of vocabulary) gestures apart from the 20 main gesture categories.…”

Section: Dataset Detailsmentioning

confidence: 99%

GeThR-Net: A Generalized Temporally Hybrid Recurrent Neural Network for Multimodal Information Fusion

Gandhi¹,

Sharma²,

Biswas³

et al. 2016

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Ye et al [50] 64.0% Wang et al [44] 85.9% Jhuo et al [12] 64.0% Tran et al [40] 86.7% Ma et al [27] 63.4% Simonyan et al [33] 88.0%…”

Section: Ucf-101mentioning

confidence: 99%

Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification

Wang

Jiang

et al. 2015

Proceedings of the 23rd ACM International Conference on Multimedia

412

243

View full text Add to dashboard Cite

Classifying videos according to content semantics is an important problem with a wide range of applications. In this paper, we propose a hybrid deep learning framework for video classification, which is able to model static spatial information, short-term motion, as well as long-term temporal clues in the videos. Specifically, the spatial and the short-term motion features are extracted separately by two Convolutional Neural Networks (CNN). These two types of CNN-based features are then combined in a regularized feature fusion network for classification, which is able to learn and utilize feature relationships for improved performance. In addition, Long Short Term Memory (LSTM) networks are applied on top of the two features to further model longer-term temporal clues. The main contribution of this work is the hybrid learning framework that can model several important aspects of the video data. We also show that (1) combining the spatial and the short-term motion features in the regularized fusion network is better than direct classification and fusion using the CNN with a softmax layer, and (2) the sequence-based LSTM is highly complementary to the traditional classification strategy without considering the temporal frame orders. Extensive experiments are conducted on two popular and challenging benchmarks, the UCF-101 Human Actions and the Columbia Consumer Videos (CCV). On both benchmarks, our framework achieves to-date the best reported performance: 91.3% on the UCF-101 and 83.5% on the CCV.

show abstract

“…Similarly to encoding collections of images, also videos have predominantly been encoded as the mean visual feature of the sampled frames [13,15]. Alternatively, using a Fisher Vector over several low-level video descriptors, such as SIFT, STIP and HOG, has been used [17,18,26].…”

Section: Video Event Recognitionmentioning

confidence: 99%

Event Fisher Vectors: Robust Encoding Visual Diversity of Visual Streams

Nagel¹,

Mensink²,

Snoek³

2015

Procedings of the British Machine Vision Conference 2015

View full text Add to dashboard Cite

In this paper we focus on event recognition in visual image streams. More specifically, we aim to construct a compact representation which encodes the diversity of the visual stream from just a few observations. For this purpose, we introduce the Event Fisher Vector, a Fisher Kernel based representation to describe a collection of images or the sequential frames of a video. We explore different generative models beyond the Gaussian mixture model as underlying probability distribution. First, the Student's-t mixture model which captures the heavy tails of the small sample size of a collection of images. Second, Hidden Markov Models to explicitly capture the temporal ordering of the observations in a stream. For all our models we derive analytical approximations of the Fisher information matrix, which significantly improves recognition performance. We extensively evaluate the properties of our proposed method on three recent datasets for event recognition in photo collections and web videos, leading to an efficient compact image representation which achieves state-of-the-art performance on all these datasets.

show abstract

Reduced Analytic Dependency Modeling: Robust Fusion for Visual Recognition

Cited by 25 publications

References 47 publications

GeThR-Net: A Generalized Temporally Hybrid Recurrent Neural Network for Multimodal Information Fusion

GeThR-Net: A Generalized Temporally Hybrid Recurrent Neural Network for Multimodal Information Fusion

Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification

Event Fisher Vectors: Robust Encoding Visual Diversity of Visual Streams

Contact Info

Product

Resources

About