Three-stream fusion network for first-person interaction recognition

Abstract

First-person interaction recognition is a challenging task because of unstable video conditions resulting from the camera wearers movement. For human interaction recognition from a first-person viewpoint, this paper proposes a three-stream fusion network with two main parts: three-stream architecture and three-stream correlation fusion. The three-stream architecture captures the characteristics of the target appearance, target motion, and camera egomotion. Meanwhile the three-stream correlation fusion combines the feature map of each of the three streams to consider the correlations among the target appearance, target motion, and camera ego-motion. The fused feature vector is robust to the camera movement and compensates for the noise of the camera ego-motion. Short-term intervals are modeled using the fused feature vector, and a long short-term memory(LSTM) model considers the temporal dynamics of the video. We evaluated the proposed method on two public benchmark datasets to validate the effectiveness of our approach. The experimental results show that the proposed fusion method successfully generated a discriminative feature vector, and our network outperformed all competing activity recognition methods in first-person videos where considerable camera ego-motion occurs.

Speeding Up Action Recognition Using Dynamic Accumulation of Residuals in Compressed Domain

“…TSCF, TSDF, and KRP represent Three-stream Correlation Fusion, Three-stream Deep Fusion, and Kernelized Ranked Pooling, respectively. The presented results of [12,16] are reported from [17]. It should be noted that all of the compared methods utilize raw RGB frames.…”

“…After extracting feature maps, maximum and average values are considered for the fusion step to obtain a unique feature map. In [17], the same architecture with a new correlation-based fusion approach is utilized. In these two articles, for the classification step, an LSTM network has been exploited.…”

“…As a result, our low complexity method utilizing available syntactic elements of the compressed domain can be employed for real-time applications. [15] 68.90% 59.50% LRCN(DOF) [15] 69.10% 89.00% LSTM(RGB_F) [27] 69.40% 70.00% LSTM(DOF) [27] 70.00% 91.00% TSDF(RGB_F) [16] 81.10% TSDF(DOF) [16] 86.40% KRP(RGB_F) [12] 73.80% KRP(DOF) [12] 85.70% TSCF(RGB_F) [17] 88.00% TSCF(DOF) [17] 94.40% Residuals(proposed) 69.33% 88.60%…”

Speeding Up Action Recognition Using Dynamic Accumulation of Residuals in Compressed Domain

“…After extracting feature maps, maximum and average values are considered for the fusion step to obtain a unique feature map. In [17], the same architecture with a new correlation-based fusion approach is utilized. In these two articles, for the classi cation step, a LSTM network has been exploited.…”

Visual Question Answering based on Local-Scene-Aware Referring Expression Generation

“…Over the past few years, visual question answering (VQA) has attracted substantial attention from both the computer vision and natural language processing communities [1][2][3][4][5][6][7][8]. Compared to the traditional tasks of computer vision or natural language processing, such as object detection [9], image captioning [10][11][12][13][14], tracking [15,16], face recognition [17,18], action recognition [19][20][21],…”