Attention-Based Convolution Skip Bidirectional Long Short-Term Memory Network for Speech Emotion Recognition

Zhang, Huiyun; Huang, Heming; Han, Henry

doi:10.1109/access.2020.3047395

Cited by 22 publications

(5 citation statements)

References 75 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More interestingly, activity recognition can be achieved through sensors integrated into smartphones. In [48], acceleration, gyroscope, and magnetometer data from a cyclist's smartphone were obtained through a sensor logger, then a 1D-CNN-BiLSTM model based on an attention mechanism was used to detect the cyclist's activity. Although the sensor-based gesture recognition approach can accurately recognize movements, it requires users to wear special sensors, resulting in poor user comfort.…”

Section: Wearable Sensor-based Methodsmentioning

confidence: 99%

Device-Free Wireless Sensing for Gesture Recognition Based on Complementary CSI Amplitude and Phase

Cai,

Li,

Chen

et al. 2024

Sensors

View full text Add to dashboard Cite

By integrating sensing capability into wireless communication, wireless sensing technology has become a promising contactless and non-line-of-sight sensing paradigm to explore the dynamic characteristics of channel state information (CSI) for recognizing human behaviors. In this paper, we develop an effective device-free human gesture recognition (HGR) system based on WiFi wireless sensing technology in which the complementary CSI amplitude and phase of communication link are jointly exploited. To improve the quality of collected CSI, a linear transform-based data processing method is first used to eliminate the phase offset and noise and to reduce the impact of multi-path effects. Then, six different time and frequency domain features are chosen for both amplitude and phase, including the mean, variance, root mean square, interquartile range, energy entropy and power spectral entropy, and a feature selection algorithm to remove irrelevant and redundant features is proposed based on filtering and principal component analysis methods, resulting in the construction of a feature subspace to distinguish different gestures. On this basis, a support vector machine-based stacking algorithm is proposed for gesture classification based on the selected and complementary amplitude and phase features. Lastly, we conduct experiments under a practical scenario with one transmitter and receiver. The results demonstrate that the average accuracy of the proposed HGR system is 98.3% and that the F1-score is over 97%.

show abstract

Section: Wearable Sensor-based Methodsmentioning

confidence: 99%

Device-Free Wireless Sensing for Gesture Recognition Based on Complementary CSI Amplitude and Phase

Cai,

Li,

Chen

et al. 2024

Sensors

View full text Add to dashboard Cite

show abstract

“…Zhang proposed a deep-learning acoustic model which used attention mechanism. It used spatiotemporal information and captured emotion-related features more effectively [31]. Feng combines word with part of speech, position and dependency syntax separately to form 3 new combined features and proposed a novel sentiment analysis model based on multichannel convolutional neural network with multi-head attention mechanism (MCNN-MA) [32].…”

Section: Attention Mechanismmentioning

confidence: 99%

Multi-Head Self-Attention Gated-Dilated Convolutional Neural Network for Word Sense Disambiguation

Zhang

2023

IEEE Access

View full text Add to dashboard Cite

Word sense disambiguation (WSD) is to determine correct sense of ambiguous word based on its context. WSD is widely used in text classification, machine translation and information retrieval and so on. In order to improve accuracy of simplified Chinese WSD, a WSD model based on multi-head selfattention and gated-dilated convolutional neural network(AGDCNN) is proposed. Ambiguous word is viewed as the center and 4 adjacent lexical units are extracted successively toward the left and right side. Words, parts of speech, and semantic categories in 4 adjacent lexical units are vectorized and the vectorized results are input into gated-dilated convolutional neural network to get discriminative features. Then, multihead self-attention is adopted to learn the difference and connection among discriminative features fully. Finally, classification weights are output from adaptive average pooling layer. Experiments are conducted on SemEval-2007: Task#5 and SemEval-2021: Task#2. Experimental results show that AGDCNN model has higher accuracy compared with other methods. Our goal is to improve the quality of simplified Chinese WSD as much as possible based on current linguistic resources and machine learning methods. The challenge we face is to extract effective discriminative features and design disambiguation model in high quality. Our novelty lies in that gated-dilated convolution is combined with multi-head self-attention to extract effective discriminative features, and learn their difference and connection from word form, parts of speech, and semantic categories.

show abstract

“…However, some of these depend on the sequential operation of the RNNs which increases the need for resources yet the devices are resource constrained. In order to consider long-term dependencies and context in speech for emotional state prediction, the models proposed in [14], [15], [16] and [17] use attention mechanisms in combination with either LSTM or bidirectional LSTM (BiLSTM) [18]. In [19] the bidirectional gated recurrent unit (BiGRU) is used to model the long-term dependencies for SER.…”

Section: Related Workmentioning

confidence: 99%

“…In [19] the bidirectional gated recurrent unit (BiGRU) is used to model the long-term dependencies for SER. Particularly authors of [16] and [17] improve the performance of SER systems by considering spatial cues through the use of convolution layers in combination with recurrent neural networks. However, as suggested in [20] though RNNs achieve promising results, they encounter problems in convergence, sluggish training that uses a lot of memory resources due to the sequential manner in which they operate.…”

Section: Related Workmentioning

confidence: 99%

Attention-Based Multi-Learning Approach for Speech Emotion Recognition With Dilated Convolution

2022

View full text Add to dashboard Cite

The success of deep learning in speech emotion recognition has led to its application in resource-constrained devices. It has been applied in human-to-machine interaction applications like social living assistance, authentication, health monitoring and alertness systems. In order to ensure a good user experience, robust, accurate and computationally efficient deep learning models are necessary. Recurrent neural networks (RNN) like long short-term memory (LSTM), gated recurrent units (GRU) and their variants that operate sequentially are often used to learn time series sequences of the signal, analyze longterm dependencies and the contexts of the utterances in the speech signal. However, due to their sequential operation, they encounter problems in convergence and sluggish training that uses a lot of memory resources and encounters the vanishing gradient problem. In addition, they do not consider spatial cues that may exist in the speech signal. Therefore, we propose an attention-based multi-learning model (ABMD) that uses residual dilated causal convolution (RDCC) blocks and dilated convolution (DC) layers with multi-head attention. The proposed ABMD model achieves comparable performance while taking global contextualized long-term dependencies between features in a parallel manner using a large receptive field with less increase in the number of parameters compared to the number of layers and considers spatial cues among the speech features. Spectral and voice quality features extracted from the raw speech signals are used as inputs. The proposed ABMD model obtained a recognition accuracy and F1 score of 93.75% and 92.50% on the SAVEE datasets, 85.89% and 85.34% on the RAVDESS datasets and 95.93% and 95.83% on the EMODB datasets. The model's robustness in terms of the confusion ratio of the individual discrete emotions especially happiness which is often confused with emotions that belong to the same dimensional plane with it also improved when validated on the same datasets.

show abstract

Attention-Based Convolution Skip Bidirectional Long Short-Term Memory Network for Speech Emotion Recognition

Cited by 22 publications

References 75 publications

Device-Free Wireless Sensing for Gesture Recognition Based on Complementary CSI Amplitude and Phase

Device-Free Wireless Sensing for Gesture Recognition Based on Complementary CSI Amplitude and Phase

Multi-Head Self-Attention Gated-Dilated Convolutional Neural Network for Word Sense Disambiguation

Attention-Based Multi-Learning Approach for Speech Emotion Recognition With Dilated Convolution

Contact Info

Product

Resources

About