A Semisupervised Tag-Transition-Based Markovian Model for Uyghur Morphology Analysis

Abstract

Morphological analysis, which includes analysis of part-of-speech (POS) tagging, stemming, and morpheme segmentation, is one of the key components in natural language processing (NLP), particularly for agglutinative languages. In this article, we investigate the morphological analysis of the Uyghur language, which is the native language of the people in the Xinjiang Uyghur autonomous region of western China. Morphological analysis of Uyghur is challenging primarily because of factors such as (1) ambiguities arising due to the likelihood of association of a multiple number of POS tags with a word stem or a multiple number of functional tags with a word suffix, (2) ambiguous morpheme boundaries, and (3) complex morphopholonogy of the language. Further, the unavailability of a manually annotated training set in the Uyghur language for the purpose of word segmentation makes Uyghur morphological analysis more difficult. In our proposed work, we address these challenges by undertaking a semisupervised approach of learning a Markov model with the help of a manually constructed dictionary of “suffix to tag” mappings in order to predict the most likely tag transitions in the Uyghur morpheme sequence. Due to the linguistic characteristics of Uyghur, we incorporate a prior belief in our model for favoring word segmentations with a lower number of morpheme units. Empirical evaluation of our proposed model shows an accuracy of about 82%. We further improve the effectiveness of the tag transition model with an active learning paradigm. In particular, we manually investigated a subset of words for which the model prediction ambiguity was within the top 20%. Manually incorporating rules to handle these erroneous cases resulted in an overall accuracy of 93.81%.

Uyghur morphological analysis using joint conditional random fields: Based on small scaled corpus

“…Therefore, this paper presents a hybrid approach that combines a small-scale tagged corpus, dictionaries, and rules. Generally speaking, one is supposed to initially use the method based on the stem and affix put forward in [14] to obtain the segmentation candidates of each word in a sentence. One may also obtain the filtering rules automatically from the manually tagged corpus and train the POS tagging model and morphological tagging model.…”

“…Y Y As a result of the complicated output results of the Uyghur morphological analysis in this research, as well as the lack of public morphological tagged corpora, there is no comparison between the experimental results and research results described in references [10] and [6]. Therefore, this paper takes the experimental results based on morphological tagging Markov model discussed in [14] as the baseline system and carries out a comparative analysis for the experimental results. The performance evaluation still adopts the standard used in [14], i.e.…”

“…Therefore, this paper takes the experimental results based on morphological tagging Markov model discussed in [14] as the baseline system and carries out a comparative analysis for the experimental results. The performance evaluation still adopts the standard used in [14], i.e. the stemming accuracy, morpheme segmentation accuracy, POS tagging accuracy, and overall accuracy.…”

“…Just as the parameter "Number of heuristic morpheme units" is used to enhance the performance of morphological tagging Markov model in [14], the parameter can influence the performance of the joint CRF model as well. Namely, the parameter "Morpheme number" is brought into the morphological tagging model.…”

“…In Uyghur morphological analysis, the accuracy of the morpheme segmentation and tagging depends largely on the POS tagging of a given word. The transition model for morphological tagging puts forward in [14] based on the stem and affix dictionaries, which are used to obtain all possible segmentation 824 GHALIP ABDUKERIM, EZIZ TURSUN, YATING YANG AND XIAO LI candidates of each Uyghur word in a training corpus (set of words). Then a Markov model for the transition probability between the stem POS and affix functional tags is created with the corresponding morphological tag sequences of segmentation candidates as the training data before finally being used to determine the most likely segmentation result.…”