Multimodal Speech Recognition with Unstructured Audio Masking

Srinivasan, Tejas; Sanabria, Ramon; Metze, Florian; Elliott, Desmond

doi:10.18653/v1/2020.nlpbt-1.2

Cited by 9 publications

(4 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…According to children learn language according to reinforcement principles by associating words with meaning (Skinner et al, 1957). Based on their research, the authors can show that language acquisition is a group based on the basic interpretation of knowledge through environmental knowledge, creating knowledge and knowledge through experience.…”

Section: Related Workmentioning

confidence: 99%

Deep Reinforcement Learning With Hidden Markov Model for Speech Recognition

Isaac,

Haruna,

Ahmad

et al. 2023

J. tech. innov.

View full text Add to dashboard Cite

Nowadays, many applications uses speech recognition especially the field of computer science and electronics, Speech Recognition (SR) is the interpretation of words spoken into a text. It is also known as Speech-To-Text (STT) or Automatic-Speech-Recognition(ASR), or just Word-Recognition(WR). The Hidden-Markov-Model (HMM) is a type of Markov model, which means that the future state of the model depends on the current state, not on the entire history of the system and the goal of HMM is to learn a sequence of hidden states from a set of known states. The Long-Short-Time-Memory (LSTM) network is a type of Recurrent Neural Network (RNN) that can learn long-term dependencies between time steps of sequence data. The LSTM network is trained by the network in order to predict the values of subsequent time steps in a series-to-series regression. Deep Neural Network (DNN) models are better classifiers than Gaussian Mixture Models (GMMs), they can generalize much better with a smaller number of parameters over complex distributions. They model distributions of different classes jointly, called “distributed” learning, or, more properly “tied” learning. This work is aimed at developing a speech recognition model that will predict isolated speech of some selected fruits in Hausa, Igbo and Yoruba language by using the predicting power of Mel-Frequency-Cepstral-Coefficient (MFCC), LSTM and HMM algorithms. The findings of the study would improve the development of better automatic speech applications systems and would benefit the academic and research community in the field of Natural Language Processing.

show abstract

Section: Related Workmentioning

confidence: 99%

Deep Reinforcement Learning With Hidden Markov Model for Speech Recognition

Isaac,

Haruna,

Ahmad

et al. 2023

J. tech. innov.

View full text Add to dashboard Cite

show abstract

“…We explore a knowledge adapter to address this challenge, which can be used as a plugin without changing the original SLU structure. Inspired by Srinivasan et al (2020), we adopt the hierarchical attention fusion mechanism (Luong, Pham, and Manning 2015;Libovický and Helcl 2017) as the knowledge adapter, which has the advantage of dynamically considering relevant supporting information for different words. Specifically, given the query vector q and the corresponding supporting information…”

Section: Multi-level Knowledge Adaptermentioning

confidence: 99%

Text Is No More Enough! A Benchmark for Profile-Based Spoken Language Understanding

Qin

Chen

et al. 2022

AAAI

View full text Add to dashboard Cite

Current researches on spoken language understanding (SLU) heavily are limited to a simple setting: the plain text-based SLU that takes the user utterance as input and generates its corresponding semantic frames (e.g., intent and slots). Unfortunately, such a simple setting may fail to work in complex real-world scenarios when an utterance is semantically ambiguous, which cannot be achieved by the text-based SLU models. In this paper, we first introduce a new and important task, Profile-based Spoken Language Understanding (ProSLU), which requires the model that not only relies on the plain text but also the supporting profile information to predict the correct intents and slots. To this end, we further introduce a large-scale human-annotated Chinese dataset with over 5K utterances and their corresponding supporting profile information (Knowledge Graph (KG), User Profile (UP), Context Awareness (CA)). In addition, we evaluate several state-of-the-art baseline models and explore a multi-level knowledge adapter to effectively incorporate profile information. Experimental results reveal that all existing text-based SLU models fail to work when the utterances are semantically ambiguous and our proposed framework can effectively fuse the supporting information for sentence-level intent detection and token-level slot filling. Finally, we summarize key challenges and provide new points for future directions, which hopes to facilitate the research.

show abstract

Section: Multi-level Knowledge Adaptermentioning

confidence: 99%

Text is no more Enough! A Benchmark for Profile-based Spoken Language Understanding

Xu¹,

Qin²,

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

Current researches on spoken language understanding (SLU) heavily are limited to a simple setting: the plain text-based SLU that takes the user utterance as input and generates its corresponding semantic frames (e.g., intent and slots). Unfortunately, such a simple setting may fail to work in complex real-world scenarios when an utterance is semantically ambiguous, which cannot be achieved by the text-based SLU models. In this paper, we first introduce a new and important task, Profile-based Spoken Language Understanding (PROSLU), which requires the model that not only relies on the plain text but also the supporting profile information to predict the correct intents and slots. To this end, we further introduce a large-scale human-annotated Chinese dataset with over 5K utterances and their corresponding supporting profile information (Knowledge Graph (KG), User Profile (UP), Context Awareness (CA)). In addition, we evaluate several state-of-the-art baseline models and explore a multi-level knowledge adapter to effectively incorporate profile information. Experimental results reveal that all existing text-based SLU models fail to work when the utterances are semantically ambiguous and our proposed framework can effectively fuse the supporting information for sentence-level intent detection and token-level slot filling. Finally, we summarize key challenges and provide new points for future directions, which hopes to facilitate the research.

show abstract

Multimodal Speech Recognition with Unstructured Audio Masking

Cited by 9 publications

References 24 publications

Deep Reinforcement Learning With Hidden Markov Model for Speech Recognition

Deep Reinforcement Learning With Hidden Markov Model for Speech Recognition

Text Is No More Enough! A Benchmark for Profile-Based Spoken Language Understanding

Text is no more Enough! A Benchmark for Profile-based Spoken Language Understanding

Contact Info

Product

Resources

About