Toward an Automatic Quality Assessment of Voice-Based Telemedicine Consultations: A Deep Learning Approach

Abstract

Maintaining a high quality of conversation between doctors and patients is essential in telehealth services, where efficient and competent communication is important to promote patient health. Assessing the quality of medical conversations is often handled based on a human auditory-perceptual evaluation. Typically, trained experts are needed for such tasks, as they follow systematic evaluation criteria. However, the daily rapid increase of consultations makes the evaluation process inefficient and impractical. This paper investigates the automation of the quality assessment process of patient–doctor voice-based conversations in a telehealth service using a deep-learning-based classification model. For this, the data consist of audio recordings obtained from Altibbi. Altibbi is a digital health platform that provides telemedicine and telehealth services in the Middle East and North Africa (MENA). The objective is to assist Altibbi’s operations team in the evaluation of the provided consultations in an automated manner. The proposed model is developed using three sets of features: features extracted from the signal level, the transcript level, and the signal and transcript levels. At the signal level, various statistical and spectral information is calculated to characterize the spectral envelope of the speech recordings. At the transcript level, a pre-trained embedding model is utilized to encompass the semantic and contextual features of the textual information. Additionally, the hybrid of the signal and transcript levels is explored and analyzed. The designed classification model relies on stacked layers of deep neural networks and convolutional neural networks. Evaluation results show that the model achieved a higher level of precision when compared with the manual evaluation approach followed by Altibbi’s operations team.

Enhancing speech emotion recognition with deep learning using multi-feature stacking and data augmentation

“…The process to obtain a Mel-spectogram is that the spectrogram is derived by short-term Fourier transform and then the resulting spectrogram is transformed into the human perception scale [16]. In order to extract spectrogram characteristics from an audio signal, it must first be divided into brief overlapping windowing, transformed into the frequency domain using the Fourier transform, and then generated into an envelope spectrogram using a Mel filter bank (see Figure 2) [17]. MFCC, which stands for Mel frequency spectrum per frame, is a type of signal spectrum that can be obtained collectively [13].…”

CUSCO: An Unobtrusive Custom Secure Audio-Visual Recording System for Ambient Assisted Living

“…Data sets comprising recordings of clinician–patient dialogues have been used for assessing communication performance and the factors that affect it in medical consultations. Audio recordings have been used in the healthcare domain for content analysis, for assessing the effectiveness of face-to-face to telephone and video consultations [ 18 , 19 ], and for detecting Alzheimer’s dementia [ 20 ] through analysis of prosodic and paralinguistic features of spontaneous speech, among other applications. The combination of audio and visual data is often used in academic assessments of the performance of medical students and trainee general practitioners (GPs) in medical consultations, and in research on communication in healthcare, such as research on the impact of the introduction of electronic health records in GPs’ offices by monitoring the GP’s focus of attention [ 21 ].…”

An Evolutionary-Based Sentiment Analysis Approach for Enhancing Government Decisions during COVID-19 Pandemic: The Case of Jordan

“…On the other hand, the F-measure is the harmonic mean of precision and recall [61], which is calculated by the following equation:…”

“…where the Precision is the percentage of correct predictions for a class relative to all the predictions of the same class [61], and the Recall is the percentage of correct predictions for a class relative to all instances that actually belong to the class [62].…”