On Word Error Rate Definitions and Their Efficient Computation for Multi-Speaker Speech Recognition Systems

Abstract

MeetEval is an open-source toolkit to evaluate all kinds of meeting transcription systems. It provides a unified interface for the computation of commonly used Word Error Rates (WERs), specifically cpWER, ORC WER and MIMO WER along other WER definitions. We extend the cpWER computation by a temporal constraint to ensure that only words are identified as correct when the temporal alignment is plausible. This leads to a better quality of the matching of the hypothesis string to the reference string that more closely resembles the actual transcription quality, and a system is penalized if it provides poor time annotations. Since word-level timing information is often not available, we present a way to approximate exact wordlevel timings from segment-level timings (e.g., a sentence) and show that the approximation leads to a similar WER as a matching with exact word-level annotations. At the same time, the time constraint leads to a speedup of the matching algorithm, which outweighs the additional overhead caused by processing the time stamps.

Environment-Aware Knowledge Distillation for Improved Resource-Constrained Edge Speech Recognition

“…where S, D, I, and C are the number of substitutions, deletions, insertions, and correct words in the estimated sequence, respectively, and N is the number of words in the true sequence (N = S + D + C). The WER is based on the Levenshtein distance and a smaller value signifies a closer approximation between the estimated word sequence and the ground truth transcription [34]. It is important to notice that, while the WER is usually presented as a value typically between 0 and 100, it does not represent a true percentage.…”