Improving Automatic Speech Recognition for Non-native English with Transfer Learning and Language Model Decoding

Sullivan, Peter; Shibano, Toshiko; Abdul-Mageed, Muhammad

doi:10.1007/978-3-031-11035-1_2

Cited by 5 publications

(1 citation statement)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, the absence of clear auditory punctuation cues necessitates that LLMs infer sentence boundaries and pauses contextually, a task fraught with potential for error [4]. Previous research has highlighted the complexities involved in speech recognition tasks, noting that even state-of-the-art models struggle with certain phonetic complexities and punctuation inference [5], [6]. The findings from this study will contribute to the existing body of knowledge by providing a comparative analysis of two leading multimodal models, offering insights into their strengths and limitations.…”

Section: Introductionmentioning

confidence: 99%

Assessing Audio Hallucination in Large Multimodal Models

Hanamaki,

Kirishima,

Narumi

2024

Preprint

View full text Add to dashboard Cite

Speech recognition systems have become increasingly integral in various applications, from virtual assistants to automated transcription services, necessitating the development of models capable of accurately processing and transcribing spoken language. The introduction of multimodal models like ChatGPT-4 and Gemini 1.5 Flash represents a significant advancement in this field, yet challenges such as audio hallucination, pronunciation handling, and punctuation placement remain critical hurdles. This study provides a comprehensive evaluation of ChatGPT-4 and Gemini 1.5 Flash, focusing on their performance in accurately transcribing English audio inputs under varying conditions. By employing rigorous statistical and qualitative analysis, including metrics like Word Error Rate (WER) and Character Error Rate (CER), the study reveals that ChatGPT-4 exhibits superior accuracy and reliability in handling complex speech patterns. Detailed examination of pronunciation handling and punctuation accuracy further elucidates the specific areas where each model excels or faces challenges. The findings demonstrate the importance of continuous refinement and enhancement of multimodal models to improve their practical applicability and reliability in real-world scenarios. This research contributes valuable insights into the strengths and limitations of leading speech recognition technologies, providing a benchmark for future developments in the field.

show abstract

Section: Introductionmentioning

confidence: 99%

Assessing Audio Hallucination in Large Multimodal Models

Hanamaki,

Kirishima,

Narumi

2024

Preprint

View full text Add to dashboard Cite

show abstract

CAPTuring Accents: An Approach to Personalize Pronunciation Training for Learners with Different L1 Backgrounds

Khaustova,

Pyshkin,

Khaustov

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Salinity Modeling Using Deep Learning with Data Augmentation and Transfer Learning

He²,

Hoang³

et al. 2023

Water

View full text Add to dashboard Cite

Salinity management in estuarine systems is crucial for developing effective water-management strategies to maintain compliance and understand the impact of salt intrusion on water quality and availability. Understanding the temporal and spatial variations of salinity is a keystone of salinity-management practices. Process-based numerical models have been traditionally used to estimate the variations in salinity in estuarine environments. Advances in data-driven models (e.g., deep learning models) make them effective and efficient alternatives to process-based models. However, a discernible research gap exists in applying these advanced techniques to salinity modeling. The current study seeks to address this gap by exploring the innovative use of deep learning with data augmentation and transfer learning in salinity modeling, exemplified at 23 key salinity locations in the Sacramento–San Joaquin Delta which is the hub of the water-supply system of California. Historical, simulated (via a hydrodynamics and water quality model), and perturbed (to create a range of hydroclimatic and operational scenarios for data-augmentation purposes) flow, and salinity data are used to train a baseline multi-layer perceptron (MLP) and a deep learning Residual Long-Short-Term Memory (Res-LSTM) network. Four other deep learning models including LSTM, Residual Network (ResNet), Gated Recurrent Unit (GRU), and Residual GRU (Res-GRU) are also examined. Results indicate that models pre-trained using augmented data demonstrate improved performance over models trained from scratch using only historical data (e.g., median Nash–Sutcliffe efficiency increased from around 0.5 to above 0.9). Moreover, the five deep learning models further boost the salinity estimation performance in comparison with the baseline MLP model, though the performance of the latter is acceptable. The models trained using augmented data are then (a) used to develop a web-based Salinity Dashboard (Dashboard) tool that allows the users (including those with no machine learning background) to quickly screen multiple management scenarios by altering inputs and visualizing the resulting salinity simulations interactively, and (b) transferred and adapted to estimate observed salinity. The study shows that transfer learning results more accurately replicate the observations compared to their counterparts from models trained from scratch without knowledge learned and transferred from augmented data (e.g., median Nash–Sutcliffe efficiency increased from around 0.4 to above 0.9). Overall, the study illustrates that deep learning models, particularly when pre-trained using augmented data, are promising supplements to existing process-based models in estuarine salinity modeling, while the Dashboard enables user engagement with those pre-trained models to inform decision-making efficiently and effectively.

show abstract

Improving Automatic Speech Recognition for Non-native English with Transfer Learning and Language Model Decoding

Cited by 5 publications

References 41 publications

Assessing Audio Hallucination in Large Multimodal Models

Assessing Audio Hallucination in Large Multimodal Models

CAPTuring Accents: An Approach to Personalize Pronunciation Training for Learners with Different L1 Backgrounds

Salinity Modeling Using Deep Learning with Data Augmentation and Transfer Learning

Contact Info

Product

Resources

About