Deep Multi-Kernel Convolutional LSTM Networks and an Attention-Based Mechanism for Videos

Abstract

Action recognition greatly benefits motion understanding in video analysis. Recurrent networks such as long short-term memory (LSTM) networks are a popular choice for motion-aware sequence learning tasks. Recently, a convolutional extension of LSTM was proposed, in which input-to-hidden and hidden-to-hidden transitions are modeled through convolution with a single kernel. This implies an unavoidable trade-off between effectiveness and efficiency. Herein, we propose a new enhancement to convolutional LSTM networks that supports accommodation of multiple convolutional kernels and layers. This resembles a Network-in-LSTM approach, which improves upon the aforementioned concern. In addition, we propose an attention-based mechanism that is specifically designed for our multi-kernel extension. We evaluated our proposed extensions in a supervised classification setting on the UCF-101 and Sports-1M datasets, with the findings showing that our enhancements improve accuracy. We also undertook qualitative analysis to reveal the characteristics of our system and the convolutional LSTM baseline.Winston H. Hsu (S'03M'07SM'12) received the Ph.D. degree in electrical engineering from Columbia University, New York, NY, USA. He is keen to realizing advanced researches towards business deliverables via academia-industry collaborations and co-founding startups. Since 2007, he

Time series forecasting of Covid-19 using deep learning models: India-USA comparative case study

“…Mathematical formulation of ConvLSTM is shown in Eqs. (19 )– (23) , where ‘*’ is Convolutional operator and ‘ · ’ represents Hadamard product [ 34, 35 ].

Fig.

…”

Deep convolutional long short-term memory for forecasting wind speed and direction

“…where • means hadamard product, and W means weight [4,11,17]. Deep fully connected-LSTM (DFC-LSTM), which is a multivariate type of LSTM that uses multiple LSTM layers, was applied in this study as it is easy to learn with the FC layer, and learns temporal features [18,19].…”

“…In CLSTM, the matrix product "•" of the LSTM is a substituted convolutional operation " * ". The equation of CLSTM is shown in [4,11,17,19].…”

Improved Soccer Action Spotting using both Audio and Video Streams

“…Advanced methods for temporal integration use particular neural network architectures, such as Convolution Neural Network (CNN) [58] or Recurrent Neural Network (RNN) [52]. More particularly, LSTM architectures are often chosen for motion-aware sequence learning tasks, which is beneficial for activity recognition [2,4]. Attention models are also harnessed to better integrate spatio-temporal information.…”