Analysis of Myanmar Word boundary and segmentation by using Statistical Approach

Abstract

This paper proposed a unified approach for Myanmar Word analysis using Finite State Automata (FSA), Rule Based Heuristic Approach and Statistical Approach. Myanmar has no inter-word space and it make the tokenizing task difficulties. Therefore, to recognize the word, we implement with FSA. Segmentation is a major problem because of no delimiter. If there were errors in segmentation, this will cause subsequence failure in further NLP processes. Segmentation is also an essential preprocessing task for Natural Language Processing, such as Machine Translation, Information Retrieval etc. In this system, the Rule Based Heuristic Approach and Statistical Approach are used with corpus based dictionary. Evaluation results showed that the method is very effective for the Myanmar language.