From phoneme to morpheme — another verification in English and Chinese using corpora —

Abstract

Zellig S. Harris's hypothesis presented in his article ÔÔFrom phoneme to morphemeÕÕ is tested; that is, how much of morpheme/word boundaries can be detected from changes in the complexity of phoneme sequences. We re-formulate his hypothesis from a more information theoretic viewpoint and use a corpus to test whether the hypothesis holds. We found that his hypothesis holds for morphemes, with an F-score of about 80%, in both English and Chinese. However, we obtained contrary results for English and Chinese with regard to word boundaries; this reflects a difference in the nature of the two languages.