Transformers in Action Recognition: A Review on Temporal Modeling

Shabaninia, Elham; Nezamabadi‐pour, Hossein; Shafizadegan, Fatemeh

doi:10.48550/arxiv.2302.01921

Cited by 5 publications

(5 citation statements)

References 248 publications

(320 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, the effects of different spatio-temporal convolutions have been investigated and 3D CNNs outperformed 2D CNNs in the concept of residual learning [36]. The use of 3D filters in combination with other techniques, such as motion-based features and transformers, has improved the accuracy and efficiency of action recognition [37]- [41]. 3D CNNs have the ability to extract distinctive features in spatial and temporal dimensions, requiring simultaneous processing of both types of features.…”

Section: Related Workmentioning

confidence: 99%

“…Although hybrid techniques have the potential to offer various advantages, there remains a lack of research investigating their complete capabilities. Furthermore, there is a lack of extensive exploration regarding the most effective methods for integrating these approaches, highlighting the necessity for additional investigation and experimentation in this domain [41].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Integration of Feature and Decision Fusion With Deep Learning Architectures for Video Classification

Kiziltepe,

Gan,

Escobar

2024

IEEE Access

View full text Add to dashboard Cite

Information fusion is frequently employed to integrate diverse inputs, including sensory data, features, or decisions, in order to leverage the advantageous relationships among various features and classifiers. This paper presents a novel approach for video classification using deep learning architectures, including ConvLSTM and vision transformer based fusion architectures, which incorporates the combination of spatial and temporal features, along with the utilisation of decision fusion at multiple levels. The proposed vision transformer based method uses a 3D CNN to extract spatio-temporal information and different attention mechanisms to pay attention to essential features for action recognition and thus learns spatiotemporal dependencies effectively. The effectiveness of the methods proposed in this paper is validated through empirical evaluations conducted on two well-known video classification datasets, namely UCF-101 and KTH. The experimental findings indicate that the utilisation of both spatial and temporal features is essential, with the superior performance gained by using temporal features as the primary source of features in conjunction with two types of distinct spatial features when compared to other configurations. The multilevel decision fusion approach proposed in this study produces results comparable to those of feature fusion methods while offering the advantage of reduced memory requirements and computational costs. The fusion of RGB, HOG, and optical flow representations has demonstrated the best performance compared to other fusion methods examined in this study. It has also been demonstrated that the vision transformer based approaches significantly outperformed the ConvLSTM based approaches. Furthermore, an ablation study was conducted to compare the performances of vision transformer based feature fusion approaches for enhancing the performance of video classification.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Integration of Feature and Decision Fusion With Deep Learning Architectures for Video Classification

Kiziltepe,

Gan,

Escobar

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…The temporality of network conditions and their evolution finds a match in GTMs [296], which can excel in processing sequential data, translating it into a coherent narrative of how 6G network conditions might evolve in response to changes in the environment. This time-series interpretation offers network operators foresight, allowing for timely interventions and adjustments.…”

Section: ) Temporal Modeling and Predictionmentioning

confidence: 99%

At the Dawn of Generative AI Era: A Tutorial-cum-Survey on New Frontiers in 6G Wireless Intelligence

Celik,

Eltawil

2024

IEEE Open J. Commun. Soc.

View full text Add to dashboard Cite

As we transition from the 5G epoch, a new horizon beckons with the advent of 6G, seeking a profound fusion with novel communication paradigms and emerging technological trends, bringing once-futuristic visions to life along with added technical intricacies. Although analytical models lay the foundations and offer systematic insights, we have recently witnessed a noticeable surge in research suggesting machine learning (ML) and artificial intelligence (AI) can efficiently deal with complex problems by complementing or replacing model-based approaches. The majority of data-driven wireless research leans heavily on discriminative AI (DAI) that requires vast real-world datasets. Unlike the DAI, Generative AI (GenAI) pertains to generative models (GMs) capable of discerning the underlying data distribution, patterns, and features of the input data. This makes GenAI a crucial asset in wireless domain wherein real-world data is often scarce, incomplete, costly to acquire, and hard to model or comprehend. With these appealing attributes, GenAI can replace or supplement DAI methods in various capacities.Accordingly, this combined tutorial-survey paper commences with preliminaries of 6G and wireless intelligence by outlining candidate 6G applications and services, presenting a taxonomy of state-of-theart DAI models, exemplifying prominent DAI use cases, and elucidating the multifaceted ways through which GenAI enhances DAI. Subsequently, we present a tutorial on GMs by spotlighting seminal examples such as generative adversarial networks, variational autoencoders, flow-based GMs, diffusion-based GMs, generative transformers, large language models, autoregressive GMs, to name a few. Contrary to the prevailing belief that GenAI is a nascent trend, our exhaustive review of approximately 120 technical papers demonstrates the scope of research across core wireless research areas, including 1) physical layer design; 2) network optimization, organization, and management; 3) network traffic analytics; 4) crosslayer network security; and 5) localization & positioning. Furthermore, we outline the central role of GMs in pioneering areas of 6G network research, including semantic communications, integrated sensing and communications, THz communications, extremely large antenna arrays, near-field communications, digital twins, AI-generated content services, mobile edge computing and edge AI, adversarial ML, and trustworthy AI. Lastly, we shed light on the multifarious challenges ahead, suggesting potential strategies and promising remedies. Given its depth and breadth, we are confident that this tutorial-cum-survey will serve as a pivotal reference for researchers and professionals delving into this dynamic and promising domain.

show abstract

“…The temporality of network conditions and their evolution finds a match in GTMs [293], which can excel in processing sequential data, translating it into a coherent narrative of how 6G network conditions might evolve in response to changes in the environment. This time-series interpretation offers network operators foresight, allowing for timely interventions and adjustments.…”

Section: ) Temporal Modeling and Predictionmentioning

confidence: 99%

At the Dawn of Generative AI Era: A Tutorial-cum-Survey on New Frontiers in 6G Wireless Intelligence

Celik,

Eltawil

2024

Preprint

View full text Add to dashboard Cite

As we transition from the 5G epoch, a new horizon beckons with the advent of 6G, seeking a profound fusion with novel communication paradigms and emerging technological trends, bringing once-futuristic visions to life along with added technical intricacies. Although analytical models lay the foundations and offer systematic insights, we have recently witnessed a noticeable surge in research suggesting machine learning (ML) and artificial intelligence (AI) can efficiently deal with complex problems by complementing or replacing model-based approaches. The majority of data-driven wireless research leans heavily on discriminative AI (DAI) that requires vast real-world datasets. Unlike the DAI, Generative AI (GenAI) pertains to generative models (GMs) capable of discerning the underlying data distribution, patterns, and features of the input data. This makes GenAI a crucial asset in wireless domain wherein real-world data is often scarce, incomplete, costly to acquire, and hard to model or comprehend. With these appealing attributes, GenAI can replace or supplement DAI methods in various capacities. Accordingly, this combined tutorial-survey paper commences with preliminaries of 6G and wireless intelligence by outlining candidate 6G applications and services, presenting a taxonomy of state-of-the-art DAI models, exemplifying prominent DAI use cases, and elucidating the multifaceted ways through which GenAI enhances DAI. Subsequently, we present a tutorial on GMs by spotlighting seminal examples such as generative adversarial networks, variational autoencoders, flow-based GMs, diffusion-based GMs, generative transformers, large language models, autoregressive GMs, to name a few. Contrary to the prevailing belief that GenAI is a nascent trend, our exhaustive review of approximately 120 technical papers demonstrates the scope of research across core wireless research areas, including 1) physical layer design; 2) network optimization, organization, and management; 3) network traffic analytics; 4) cross-layer network security; and 5) localization & positioning. Furthermore, we outline the central role of GMs in pioneering areas of 6G network research, including semantic communications, integrated sensing and communications, THz communications, extremely large antenna arrays, near-field communications, digital twins, AI-generated content services, mobile edge computing and edge AI, adversarial ML, and trustworthy AI. Lastly, we shed light on the multifarious challenges ahead, suggesting potential strategies and promising remedies. Given its depth and breadth, we are confident that this tutorial-cum-survey will serve as a pivotal reference for researchers and professionals delving into this dynamic and promising domain.

show abstract

Transformers in Action Recognition: A Review on Temporal Modeling

Cited by 5 publications

References 248 publications

Integration of Feature and Decision Fusion With Deep Learning Architectures for Video Classification

Integration of Feature and Decision Fusion With Deep Learning Architectures for Video Classification

At the Dawn of Generative AI Era: A Tutorial-cum-Survey on New Frontiers in 6G Wireless Intelligence

At the Dawn of Generative AI Era: A Tutorial-cum-Survey on New Frontiers in 6G Wireless Intelligence

Contact Info

Product

Resources

About