End-to-End Lip-Reading Open Cloud-Based Speech Architecture

Abstract

Deep learning technology has encouraged research on noise-robust automatic speech recognition (ASR). The combination of cloud computing technologies and artificial intelligence has significantly improved the performance of open cloud-based speech recognition application programming interfaces (OCSR APIs). Noise-robust ASRs for application in different environments are being developed. This study proposes noise-robust OCSR APIs based on an end-to-end lip-reading architecture for practical applications in various environments. Several OCSR APIs, including Google, Microsoft, Amazon, and Naver, were evaluated using the Google Voice Command Dataset v2 to obtain the optimum performance. Based on performance, the Microsoft API was integrated with Google’s trained word2vec model to enhance the keywords with more complete semantic information. The extracted word vector was integrated with the proposed lip-reading architecture for audio-visual speech recognition. Three forms of convolutional neural networks (3D CNN, 3D dense connection CNN, and multilayer 3D CNN) were used in the proposed lip-reading architecture. Vectors extracted from API and vision were classified after concatenation. The proposed architecture enhanced the OCSR API average accuracy rate by 14.42% using standard ASR evaluation measures along with the signal-to-noise ratio. The proposed model exhibits improved performance in various noise settings, increasing the dependability of OCSR APIs for practical applications.

Multimodal audiovisual speech recognition architecture using a three‐feature multi‐fusion method for noise‐robust systems

“…To evaluate the proposed model, three features were combined to compare the training step, error rate, and accuracy, and SNR evaluation was performed by synthesizing speeches in nine noise scenario environments that could be used in a real environment. Consequently, the proposed model exhibited the most stable convergence process and superior performance without significant changes in parameters and training time compared to other models, and it proved to be more robust to noise than previous studies [30,34]. In particular, by adding a new specific point, the log-Mel spectrogram information, the model exhibited a performance improvement of approximately 1.6%-1.2% over the performance of the model that used only two existing features (word embedding and lip movement).…”

“…The proposed audio recognition module is illustrated in Figure 3. In previous studies [34,35], we used an open cloud-based speech recognition API using Microsoft's Azure Cognitive Services API [19], which had approximately 5%-10% better word recognition rates than Google Assistant and Amazon Transcribe. To mitigate the impact of performance changes over time, a new API that surpasses the current API is provided whenever available.…”

“…Additionally, following the "transition" structure shown in Figure S1A(b) (Table S1), a standard dropout layer was connected, and pixels were randomly dropped to prevent strong correlation in feature maps between successive frames [39]. In addition, the spatial dropout layer connected to the "transition" structure, shown in Figure S2C, was effectively used to extract fine movement features such as lips, teeth, and tongue with strong spatial correlation [31][32][33][34][35][36][37][38][39][40][41][42][43]. Therefore, the proposed dense spatial-temporal CNN network comprises one layer that represents a nonlinear transformation Hl, and the output of the layer can be expressed as x l (3), where x 0 , x 1 , ÁÁÁ, and x (l-1) denote the volume of the 3D feature created in the previous layer and [ÁÁÁ] denotes a concatenation operation.…”

Noise-Robust Multimodal Audio-Visual Speech Recognition System for Speech-Based Interaction Applications

“…In this study, an end-to-end visual speech recognition-based interaction system for speech interaction in a virtual aquarium environment was proposed. Recognized words were vectored by combining pretrained word embedding with Microsoft API, which showed the best performance and dense dispersion in previous studies [ 45 ]. Feature vectorization was performed on the image sequence input through video via the visual processing module, and word vectorization was combined to output the predicted word.…”

“…In addition, the performance of the OCSR API is constantly updated by many companies that provide API services, and it depends on the study date and the type of training data. Therefore, we used the Microsoft Azure API, which has been proven to be superior in the results of previous studies [ 45 ]. In addition, the existing speech API can be replaced if another speech API with better performance is released and is not affected by performance changes over time.…”