Lingstat

Yamron, Jonathan; Baker, James K.; Bamberg, Paul; Chevalier, Haakon; Dietzel, Taiko; Elder, John F.; Kampmann, Frank; Mandel, Mark A.; Manganaro, Linda; Margolis, Todd P.; Steele, Elizabeth

doi:10.3115/1075671.1075714

Cited by 3 publications

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Mostly-unsupervised statistical segmentation of Japanese kanji sequences

Ando¹,

Lee²

2003

Nat. Lang. Eng.

View full text Add to dashboard Cite

Given the lack of word delimiters in written Japanese, word segmentation is generally considered a crucial first step in processing Japanese texts. Typical Japanese segmentation algorithms rely either on a lexicon and syntactic analysis or on pre-segmented data; but these are laborintensive, and the lexico-syntactic techniques are vulnerable to the unknown word problem. In contrast, we introduce a novel, more robust statistical method utilizing unsegmented training data. Despite its simplicity, the algorithm yields performance on long kanji sequences comparable to and sometimes surpassing that of state-of-the-art morphological analyzers over a variety of error metrics. The algorithm also outperforms another mostly-unsupervised statistical algorithm previously proposed for Chinese.Additionally, we present a two-level annotation scheme for Japanese to incorporate multiple segmentation granularities, and introduce two novel evaluation metrics, both based on the notion of a compatible bracket, that can account for multiple granularities simultaneously.

show abstract

Mostly-unsupervised statistical segmentation of Japanese kanji sequences

Ando¹,

Lee²

2003

Nat. Lang. Eng.

View full text Add to dashboard Cite

show abstract

Japanese word segmentation by hidden Markov model

Papageorgiou¹

1994

Proceedings of the Workshop on Human Language Technology - HLT '94

View full text Add to dashboard Cite

The processing of Japanese text is complicated by the fact that there are no word delimiters. To segment Japanese text, systems typically use knowledge-based methods and large lexicons. This paper presents a novel approach to Japanese word segmentation which avoids the need for Japanese word lexicons and explicit rule bases. The algorithm utilizes a hidden Markov model, a stochastic process, to determine word boundaries. This method has achieved 91% accuracy in segmenting words in a test corpus.

show abstract

The automatic component of the LINGSTAT machine-aided translation system

Yamron¹,

Cant²,

Demedts³

et al. 1994

Proceedings of the Workshop on Human Language Technology - HLT '94

View full text Add to dashboard Cite

We present the newest implementation of the LINGSTAT machine-aided translation system. The moat signiflcsat change from earlier versions is a new set of modules that produce a draft translation of the document for the user to refer to or modify. This paper describes these modules, with special emphasis on an automatically trained lexicalized grammar used in the parsing module. Some preHminary results from the January 1994 ARPA evaluation are reported.

show abstract

Lingstat

Cited by 3 publications

References 5 publications

Mostly-unsupervised statistical segmentation of Japanese kanji sequences

Mostly-unsupervised statistical segmentation of Japanese kanji sequences

Japanese word segmentation by hidden Markov model

The automatic component of the LINGSTAT machine-aided translation system

Contact Info

Product

Resources

About