The Conceptual Similarity of Intonational Tones and its Effects on Intertranscriber Reliability

Abstract

Tests of intertranscriber agreement in prosodically-labeled corpora have been used as an objective performance measure of reliability. Reasonably high agreements among labelers have been found, but systematic disagreements exist, indicating that some intonational patterns are more difficult for transcribers to label while others are easier. This may be due to differences in the way transcribers distinguish between tonal labels for pitch events. We developed a method to map the subjective similarity space for the categories in an intonational transcription system. From this map, we derived a conceptual tone similarity similarity index indicating the distance between tone categories. This subjective similarity index is used to predict the intertranscriber reliability. It is found that tones which are conceptually similar result in higher transcriptional disagreements while tones which are conceptually dissimilar result in lower disagreement.

Analysis of inter-transcriber consistency in the Cat_ToBI prosodic labeling system

“…Confusion matrices have been used by [17] and [21] to analyze the conceptual similarity of ToBI tones. [21] uses the confusion matrix in absolute terms while [17] introduces the equations above to compare tag assignments.…”

“…[21] uses the confusion matrix in absolute terms while [17] introduces the equations above to compare tag assignments. [17] also presents separate tables for each pair of labellers.…”

“…MDS has been already used in the context of ToBI labeling as an inter-transcriber reliability measure in [17]. In [17], MDS is used to convert into distances a categorical index named the Conceptual Similarity Index.…”

“…In [17], MDS is used to convert into distances a categorical index named the Conceptual Similarity Index. These distances are assumed to be representative of the difference in criteria between taggers and displayed in a set of 2D plots, one for every pair of labellers.…”

“…In [17] a set of transcribers are questioned about the inter-similarity of the various ToBI labels. Their answers show that, for example, they find the pair H* and L+H* the most difficult pair of symbols to separate.…”