Three-Stream Network With Bidirectional Self-Attention for Action Recognition in Extreme Low Resolution Videos

Purwanto, Didik; Pramono, Rizard Renanda Adhi; Chen, Yie‐Tarng; Fang, Wen‐Hsien

doi:10.1109/lsp.2019.2923918

Cited by 35 publications

(9 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, researchers use convolution network as the basic architecture for human action recognition [21,22]. For example, Purwanto et al [23] designed a two-way selfattention network for human action recognition, which is mainly aggregated with the three-stream network in this paper to recognize human action in ultra-low-resolution video, so as to further show the time dependence between spatial-temporal features. Zhang et al [24] used two stream convolutional neural network for feature extraction, and then added a self-attention mechanism to the action recognition framework.…”

Section: Cnn-based Methods Of Action Classification Based On Attention Mechanismmentioning

confidence: 99%

Survey of Action Recognition Based on Attention Mechanism

Yao

et al. 2021

Proceedings of 2021 Chinese Intelligent Automation Conference

View full text Add to dashboard Cite

Attention mechanism for action recognition is to enhance data by using attention mechanism so as to recognize human action in video. This research has been intensely investigated in the field of computer vision, which has attracted extensive attention from academia and industry. It is also the premise of intelligent interaction and human-computer cooperation, which helps the machine to perceive the external environment. In the past decade, especially the emergence of deep learning technology, great progress has been made in this field. Therefore, it is necessary to make a comprehensive review of the recent work. This paper summarizes the research background and application fields, typical datasets and experimental evaluation, and compares various attention mechanism for human action feature representation and action classification algorithms. Finally, we look forward to the development prospect of human action recognition in self-driving vehicles. This paper provides ideas for the follow-up research on the understanding of the behavior of self-driving vehicles, and makes self-driving vehicles become interactive wheeled robots.

show abstract

Section: Cnn-based Methods Of Action Classification Based On Attention Mechanismmentioning

confidence: 99%

Survey of Action Recognition Based on Attention Mechanism

Yao

et al. 2021

Proceedings of 2021 Chinese Intelligent Automation Conference

View full text Add to dashboard Cite

show abstract

“…Afterwards, Transformer layers are trained for a downstream task on those features. With this approach, many works [52], [67], [78], [81], [86], [95], [101], [136], [141], [171] are still able to train the Transformer on small datasets (<10k training samples). However, it is definitely common to use medium to large datasets, as in [53], [54], [56], [57], [58], [59], [66], [93], [172].…”

Section: Training Regimementioning

confidence: 99%

Video Transformers: A Survey

Selva¹,

Johansen²,

Escalera³

et al. 2022

Preprint

View full text Add to dashboard Cite

Transformer models have shown great success modeling long-range interactions. Nevertheless, they scale quadratically with input length and lack inductive biases. These limitations can be further exacerbated when dealing with the high dimensionality of video. Proper modeling of video, which can span from seconds to hours, requires handling long-range interactions. This makes Transformers a promising tool for solving video related tasks, but some adaptations are required. While there are previous works that study the advances of Transformers for vision tasks, there is none that focus on in-depth analysis of video-specific designs. In this survey we analyse and summarize the main contributions and trends for adapting Transformers to model video data. Specifically, we delve into how videos are embedded and tokenized, finding a very widspread use of large CNN backbones to reduce dimensionality and a predominance of patches and frames as tokens. Furthermore, we study how the Transformer layer has been tweaked to handle longer sequences, generally by reducing the number of tokens in single attention operation. Also, we analyse the self-supervised losses used to train Video Transformers, which to date are mostly constrained to contrastive approaches. Finally, we explore how other modalities are integrated with video and conduct a performance comparison on the most common benchmark for Video Transformers (i.e., action classification), finding them to outperform 3D CNN counterparts with equivalent FLOPs and no significant parameter increase.

show abstract

“…[15] introduce a novel spatial-temporal multi-head self-attention mechanism based on the combination of super-resolution, knowledge distillation and attention mechanisms to learn powerful yet general features. Moreover, Purwanto et Al., in [14], address the visual degradation problem with an additional input source, namely the trajectory. This architecture complements the usual spatial-temporal information with trajectory patterns capturing features robust to visual distortion.…”

Section: Related Workmentioning

confidence: 99%

“…Exploited Data Backbone MSN eLR Three 3D conv Streams Semi-Coupled [6] eLR Two 2D Conv Streams ISR [18] LR + HR Custom 2D Conv S.T.M.H.S.A. [15] eLR + HR Double I3D TSN-BSA [14] eLR Three Stream I3D Fully Coupled [29] eLR C3D Multi-Siamese [17] eLR + HR Custom 2D Conv The reported values are the average accuracies over the three official splits.…”

Section: Architecturementioning

confidence: 99%