Robust Speech Recognition via Large-Scale Weak Supervision

Radford, Alec; Kim, Jong Wook; Xu, Tao; Brockman, Greg; McLeavey, Christine; Sutskever, Ilya

doi:10.48550/arxiv.2212.04356

Cited by 162 publications

(179 citation statements)

References 0 publications

Supporting

Mentioning

172

Contrasting

Unclassified

Order By: Relevance

“…When the recordings contained speech from the investigator at the beginning and end, we trimmed the recordings. All recordings from the SS task were automatically transcribed using OpenAI's Whisper (40), that is an Automatic Speech Recognition (ASR) system trained for 680,000 h of multilingual and multitask supervised data collected from the web.…”

Section: Data Pre-processingmentioning

confidence: 99%

Multilingual evaluation of interpretable biomarkers to represent language and speech patterns in Parkinson's disease

et al. 2023

View full text Add to dashboard Cite

Motor impairments are only one aspect of Parkinson's disease (PD), which also include cognitive and linguistic impairments. Speech-derived interpretable biomarkers may help clinicians diagnose PD at earlier stages and monitor the disorder's evolution over time. This study focuses on the multilingual evaluation of a composite array of biomarkers that facilitate PD evaluation from speech. Hypokinetic dysarthria, a motor speech disorder associated with PD, has been extensively analyzed in previously published studies on automatic PD evaluation, with a relative lack of inquiry into language and task variability. In this study, we explore certain acoustic, linguistic, and cognitive information encoded within the speech of several cohorts with PD. A total of 24 biomarkers were analyzed from American English, Italian, Castilian Spanish, Colombian Spanish, German, and Czech by conducting a statistical analysis to evaluate which biomarkers best differentiate people with PD from healthy participants. The study leverages conceptual robustness as a criterion in which a biomarker behaves the same, independent of the language. Hence, we propose a set of speech-based biomarkers that can effectively help evaluate PD while being language-independent. In short, the best acoustic and cognitive biomarkers permitting discrimination between experimental groups across languages were fundamental frequency standard deviation, pause time, pause percentage, silence duration, and speech rhythm standard deviation. Linguistic biomarkers representing the length of the narratives and the number of nouns and auxiliaries also provided discrimination between groups. Altogether, in addition to being significant, these biomarkers satisfied the robustness requirements.

show abstract

Section: Data Pre-processingmentioning

confidence: 99%

Multilingual evaluation of interpretable biomarkers to represent language and speech patterns in Parkinson's disease

et al. 2023

View full text Add to dashboard Cite

show abstract

“…We hereby introduce the multilingual multitask acoustic model Whisper [44] and the SSL XLSR-53 [45] model as the pre-trained encoders to explore if cross-lingual speech representations can advance the Dutch dysarthric speech SLU. Moreover, the investigation of the crosslingual implementations in which language properties are not explicitly exploited suggests our results hold for dysarthric SLU in other languages than Dutch, though we will not formally evaluate this.…”

Section: Contributionsmentioning

confidence: 99%

Benefits of pre-trained mono- and cross-lingual speech representations for spoken language understanding of Dutch dysarthric speech

Wang

hamme

2023

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

With the rise of deep learning, spoken language understanding (SLU) for command-and-control applications such as a voice-controlled virtual assistant can offer reliable hands-free operation to physically disabled individuals. However, due to data scarcity, it is still a challenge to process dysarthric speech. Pre-training (part of) the SLU model with supervised automatic speech recognition (ASR) targets or with self-supervised learning (SSL) may help to overcome a lack of data, but no research has shown which pre-training strategy performs better for SLU on dysarthric speech and to which extent the SLU task benefits from knowledge transfer from pre-training with dysarthric acoustic tasks. This work aims to compare different mono- or cross-lingual pre-training (supervised and unsupervised) methodologies and quantitatively investigates the benefits of pre-training for SLU tasks on Dutch dysarthric speech. The designed SLU systems consist of a pre-trained speech representations encoder and a SLU decoder to map encoded features to intents. Four types of pre-trained encoders, a mono-lingual time-delay neural network (TDNN) acoustic model, a mono-lingual transformer acoustic model, a cross-lingual transformer acoustic model (Whisper), and a cross-lingual SSL Wav2Vec2.0 model (XLSR-53), are evaluated complemented with three types of SLU decoders: non-negative matrix factorization (NMF), capsule networks, and long short-term memory (LSTM) networks. The acoustic analysis of the four pre-trained encoders are tested on Dutch dysarthric home-automation data with word error rate (WER) results to investigate the correlations of the dysarthric acoustic task (ASR) and the semantic task (SLU). By introducing the intelligibility score (IS) as a metric of the impairment severity, this paper further quantitatively analyzes dysarthria-severity-dependent models for SLU tasks.

show abstract

“…Whisper is a recent voice recognition model that trained for 68,000 h on various linguistic and multipurpose supervised data acquired from the internet. Whisper achieved high accuracy in speech recognition, although all size versions that were released were not fit for MCUs [25]. Large Language Models (LLMs), which were originally developed in the context of Natural Language Processing and Understanding, have recently become very popular as foundation models for various downstream tasks [26].…”

Section: Utilizing Audio Signals For Classification Problemsmentioning

confidence: 99%

LimitAccess: on-device TinyML based robust speech recognition and age classification

Maayah

Abunada

Al-Janahi

et al. 2023

Discov Artif Intell

View full text Add to dashboard Cite

Automakers from Honda to Lamborghini are incorporating voice interaction technology into their vehicles to improve the user experience and offer value-added services. Speech recognition systems are a key component of smart cars, enhancing convenience and safety for drivers and passengers. In the future, safety-critical features may rely on speech recognition, but this raises concerns about children accessing such services. To address this issue, the LimitAccess system is proposed, which uses TinyML for age classification and helps parents limit children’s access to critical speech recognition services. This study employs a lite convolutional neural network (CNN) model for two different reasons: First, CNN showed superior accuracy compared to other audio classification models for age classification problems. Second, the lite model will be integrated into a microcontroller to meet its limited resource requirements. To train and evaluate our model, we created a dataset that included child and adult voices of the keyword “open”. The system approach categorizes voices into age groups (child, adult) and then utilizes that categorization to grant access to a car. The robustness of the model was enhanced by adding a new class (recordings) to the dataset, which enabled our system to detect replay and synthetic voice attacks. If an adult voice is detected, access to start the car will be granted. However, if a child’s voice or a recording is detected, the system will display a warning message that educates the child about the dangers and consequences of the improper use of a car. Arduino Nano 33 BLE sensing was our embedded device of choice for integrating our trained, optimized model. Our system achieved an overall F1 score of 87.7% and 85.89% accuracy. LimitAccess detected replay and synthetic voice attacks with an 88% F1 score.

show abstract

Robust Speech Recognition via Large-Scale Weak Supervision

Cited by 162 publications

References 0 publications

Multilingual evaluation of interpretable biomarkers to represent language and speech patterns in Parkinson's disease

Multilingual evaluation of interpretable biomarkers to represent language and speech patterns in Parkinson's disease

Benefits of pre-trained mono- and cross-lingual speech representations for spoken language understanding of Dutch dysarthric speech

LimitAccess: on-device TinyML based robust speech recognition and age classification

Contact Info

Product

Resources

About