Audio-visual feature fusion via deep neural networks for automatic speech recognition

Rahmani, Mohammad Hasan; Almasganj, Farshad; Seyyedsalehi, Seyyed Ali

doi:10.1016/j.dsp.2018.06.004

Cited by 28 publications

(8 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Reference [21], deep autoencoders are proposed to produce efficient bimodal features from the audio and visual stream inputs. The authors obtained an average relative reduction of 36.9% for a range of different noisy conditions, and also, a relative reduction of 19.2% for the clean condition in terms of the Phoneme Error Rates (PER) in comparison with the baseline method.…”

Section: Deep Learning Techniquesmentioning

confidence: 99%

Detecting Respiratory Pathologies Using Convolutional Neural Networks and Variational Autoencoders for Unbalancing Data

García-Ordás

Benítez-Andrades

García-Rodríguez

et al. 2020

Sensors

View full text Add to dashboard Cite

The aim of this paper was the detection of pathologies through respiratory sounds. The ICBHI (International Conference on Biomedical and Health Informatics) Benchmark was used. This dataset is composed of 920 sounds of which 810 are of chronic diseases, 75 of non-chronic diseases and only 35 of healthy individuals. As more than 88% of the samples of the dataset are from the same class (Chronic), the use of a Variational Convolutional Autoencoder was proposed to generate new labeled data and other well known oversampling techniques after determining that the dataset classes are unbalanced. Once the preprocessing step was carried out, a Convolutional Neural Network (CNN) was used to classify the respiratory sounds into healthy, chronic, and non-chronic disease. In addition, we carried out a more challenging classification trying to distinguish between the different types of pathologies or healthy: URTI, COPD, Bronchiectasis, Pneumonia, and Bronchiolitis. We achieved results up to 0.993 F-Score in the three-label classification and 0.990 F-Score in the more challenging six-class classification.

show abstract

Section: Deep Learning Techniquesmentioning

confidence: 99%

Detecting Respiratory Pathologies Using Convolutional Neural Networks and Variational Autoencoders for Unbalancing Data

García-Ordás

Benítez-Andrades

García-Rodríguez

et al. 2020

Sensors

View full text Add to dashboard Cite

show abstract

“…Feature fusion method achieves good performance in processing speech data. Hasan et al [27] proposed an audio-visual feature fusion via deep neural networks and implemented speech recognition with low error rate. In addition, the audio-visual feature fusion was used to recognize lip language [28].…”

Section: ] Summarizedmentioning

confidence: 99%

Attention-Block Deep Learning Based Features Fusion in Wearable Social Sensor for Mental Wellbeing Evaluations

et al. 2020

View full text Add to dashboard Cite

With the progressive increase of stress, anxiety and depression in working and living environment, mental health assessment becomes an important social interaction research topic. Generally, clinicians evaluate the psychology of participants through an effective psychological evaluation and questionnaires. However, these methods suffer from subjectivity and memory effects. In this paper, a new multi-sensing wearable device has been developed and applied in self-designed psychological tests. Speech under different emotions as well as behavior signals are captured and analyzed. The mental state of the participants is objectively assessed through a group of psychological questionnaires. In particular, we propose an attention-based block deep learning architecture within the device for multi-feature classification and fusion analysis. This enables the deep learning architecture to autonomously train to obtain the optimum fusion weights of different domain features. The proposed attention-based architecture has led to improving performance compared with direct connecting fusion method. Experimental studies have been carried out in order to verify the effectiveness and robustness of the proposed architecture. The obtained results have shown that the wearable multi-sensing devices equipped with the attention-based block deep learning architecture can effectively classify mental state with better performance. INDEX TERMS Mental health assessment, wearable device, attention-based feature fusion.

show abstract

“…Researchers in [12]- [14] it was shown that there is a very close relationship between the theory of error-correcting coding and the theory of neural networks, which are also often used for digital signal processing [15], [16]. This creates quite definite prospects for the practical use of multivalued logics, which are actively being developed at the present time [17], [18].…”

Section: Introductionmentioning

confidence: 99%

Algebraic fields and rings as a digital signal processing tool

Matrassulova

Vitulyova

Konshin

et al. 2022

IJEECS

View full text Add to dashboard Cite

It is shown that algebraic fields and rings can become a very promising tool for digital signal processing. This is mainly due to the fact that any digital signals change in a finite range of amplitudes and, therefore, there are only a finite set of levels that can correspond to the amplitudes of a signal reduced to a discrete form. This allows you to establish a one-to-one correspondence between the set of levels and such algebraic structures as fields, rings, etc. This means that a function that takes values in any of the algebraic structures containing a finite set of elements can serve as a model of a signal reduced to a discrete form. A special case of such a signal model are functions that take values in Galois fields. It is shown that, along with Galois fields, in certain cases, algebraic rings contain zero divisors can be used to construct signal models. This representation is convenient because in this case it becomes possible to independently operate with the digits of the number that enumerates the signal levels. A simple and intuitive method for constructing rings is proposed, based on an analogy with the method of algebraic extensions.

show abstract

Audio-visual feature fusion via deep neural networks for automatic speech recognition

Cited by 28 publications

References 14 publications

Detecting Respiratory Pathologies Using Convolutional Neural Networks and Variational Autoencoders for Unbalancing Data

Detecting Respiratory Pathologies Using Convolutional Neural Networks and Variational Autoencoders for Unbalancing Data

Attention-Block Deep Learning Based Features Fusion in Wearable Social Sensor for Mental Wellbeing Evaluations

Algebraic fields and rings as a digital signal processing tool

Contact Info

Product

Resources

About