Accuracy of Cloud-Based Speech Recognition Open Application Programming Interface for Medical Terms of Korean

Abstract

Background There are limited data on the accuracy of cloud-based speech recognition (SR) open application programming interfaces (APIs) for medical terminology. This study aimed to evaluate the medical term recognition accuracy of current available cloud-based SR open APIs in Korean. Methods We analyzed the SR accuracy of currently available cloud-based SR open APIs using real doctor–patient conversation recordings collected from an outpatient clinic at a large tertiary medical center in Korea. For each original and SR transcription, we analyzed the accuracy rate of each cloud-based SR open API (i.e., the number of medical terms in the SR transcription per number of medical terms in the original transcription). Results A total of 112 doctor–patient conversation recordings were converted with three cloud-based SR open APIs (Naver Clova SR from Naver Corporation; Google Speech-to-Text from Alphabet Inc.; and Amazon Transcribe from Amazon), and each transcription was compared. Naver Clova SR (75.1%) showed the highest accuracy with the recognition of medical terms compared to the other open APIs (Google Speech-to-Text, 50.9%, P < 0.001; Amazon Transcribe, 57.9%, P < 0.001), and Amazon Transcribe demonstrated higher recognition accuracy compared to Google Speech-to-Text ( P < 0.001). In the sub-analysis, Naver Clova SR showed the highest accuracy in all areas according to word classes, but the accuracy of words longer than five characters showed no statistical differences (Naver Clova SR, 52.6%; Google Speech-to-Text, 56.3%; Amazon Transcribe, 36.6%). Conclusion Among three current cloud-based SR open APIs, Naver Clova SR which manufactured by Korean company showed highest accuracy of medical terms in Korean, compared to Google Speech-to-Text and Amazon Transcribe. Although limitations are existing in the recognition of medical terminology, there is a lot of rooms for improvement of this promising technology by combining strengths of each SR engines.

Intelligent speech technologies for transcription, disease diagnosis, and medical equipment interactive control in smart hospitals: A review

“… Zhejiang Provincial People's Hospital Generate and extract pathological examination reports: 52h labeled pathological report recordings. ASR system with Adaptive technology Recognition rate = 77.87%; reduces labor costs; improves work efficiency and service quality [ 81 ] Western Paraná State University Audios collected from 30 volunteers Google API and Microsoft API integrated with the web Reduces the time to elaborate reports in the radiology [ 89 ] University Hospital Mannheim Lab test: 22 volunteers; Filed test: 2 male emergency physicians IBM's Via-Voice Millennium Edition version 7.0 The overall recognition rate is about 85%. About 75% in emergency medical missions [ 77 ] Kerman University of Medical Sciences Notes of hospitalized Patients from 2 groups of 35 nurses Offline SR (Nevisa) Online SR (Speechtexter) Users' technological literacy; Possibility of error report: handwritten < offline SR < online SR [ 74 ] University of North Carolina School of Medicine 6 radiologists dictated using speech-recognition software PowerScribe 360 v4.0-SP2 reporting software Near-significant increase in the rate of dictation errors; most errors are minor single incorrect words.…”