Simulation of Subjective Closed Captioning Quality Assessment Using Prediction Models

Abstract

As a primary user group, Deaf or Hard of Hearing (D/HOH) audiences use Closed Captioning (CC) service to enjoy the TV programs with audio by reading text. However, the D/HOH communities are not completely satisfied with the quality of CC even though the government regulators entail certain rules in the CC quality factors. The measure of the CC quality is often interpreted as an accuracy on translation and regulators use the empirical models to assess. The need of a subjective quality scale comes from the gap in between current empirical assessment models and the audience perceived quality. It is possible to fill the gap by including the subjective assessment by D/HOH audiences. This research proposes a design of an automatic quality assessment system for CC which can predict the D/HOH audience subjective ratings. A simulated rater is implemented based on literature and the CC quality factor representative value extraction algorithm is developed. Three prediction models are trained with a set of CC quality values and corresponding rating scores, then they are compared to find the feasible prediction model.

HATS: An Open Data Set Integrating Human Perception Applied to the Evaluation of Automatic Speech Recognition Metrics

“…Consequently, there has been a growing interest in developing new metrics to evaluate ASR systems. Some researchers [23,27,13,20,2] have therefore started exploring alternative metrics that can more accurately assess the quality and effectiveness of automatic transcriptions. Similarly, these issues have been observed in the field of machine translation.…”