Syllable-based Korean POS Tagging Based on Combining a Pre-analyzed Dictionary with Machine Learning

Lee, Chung-Hee; Lim, Joon-Ho; Lim, Soojong; Kim, Hyunki

doi:10.5626/jok.2016.43.3.362

Cited by 4 publications

(8 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, when it comes to the syllable-based system, our model did not achieve comparable performance. The syllable-based system, as outlined in the research by Lee and others [14], has been substantially enhanced through the use of a pre-analyzed dictionary. This integration has significantly elevated its performance, allowing for more accurate handling of various linguistic elements.…”

Section: Basic Performancementioning

confidence: 99%

“…To address the drawbacks of dictionary dependence, syllable-by-syllable morphological analysis has emerged as an alternative [8,9,11,13,14,17,19,[25][26][27][29][30][31]. This approach involves either tagging each syllable and then applying a base-form restoration dictionary [14,26] or tagging the syllable with the base form already restored [31]. However, a notable drawback is the difficulty in accurately identifying morpheme boundaries.…”

Section: Syllable-unit Morphological Analysismentioning

confidence: 99%

“…In recent years, Korean morphological analyses have witnessed a diverse range of methodologies [3–31]. The agglutinative nature of the Korean language poses challenges that have inspired researchers to devise innovative solutions, laying the foundation for future investigations.…”

Section: Related Workmentioning

confidence: 99%

“…While some approaches rely on rules or dictionaries to capture this understanding [12], constructing and updating dictionaries for varied text vocabularies can be challenging. As a result, methods focusing on tagging syllable units without a dictionary have been proposed [9,13,14,26] and studied for enhancement [8,11,19,25,27,[29][30][31]. Mechanically, syllable-by-syllable morphological analysis can be achieved by either tagging syllables and then applying a base-form restoration dictionary [14,26] or by tagging syllables with the base form pre-restored [31].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Transformer‐based reranking for improving Korean morphological analysis systems

Ryu,

Lim,

Kwon

et al. 2024

ETRI Journal

Self Cite

View full text Add to dashboard Cite

This study introduces a new approach in Korean morphological analysis combining dictionary‐based techniques with Transformer‐based deep learning models. The key innovation is the use of a BERT‐based reranking system, significantly enhancing the accuracy of traditional morphological analysis. The method generates multiple suboptimal paths, then employs BERT models for reranking, leveraging their advanced language comprehension. Results show remarkable performance improvements, with the first‐stage reranking achieving over 20% improvement in error reduction rate compared with existing models. The second stage, using another BERT variant, further increases this improvement to over 30%. This indicates a significant leap in accuracy, validating the effectiveness of merging dictionary‐based analysis with contemporary deep learning. The study suggests future exploration in refined integrations of dictionary and deep learning methods as well as using probabilistic models for enhanced morphological analysis. This hybrid approach sets a new benchmark in the field and offers insights for similar challenges in language processing applications.

show abstract

Section: Basic Performancementioning

confidence: 99%

Section: Syllable-unit Morphological Analysismentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Transformer‐based reranking for improving Korean morphological analysis systems

Ryu,

Lim,

Kwon

et al. 2024

ETRI Journal

Self Cite

View full text Add to dashboard Cite

show abstract

“…Using the pre-analysis eojeol dictionary (PED) [34]- [36] can overcome these problems. According to this method, a dictionary of analyzed eojeol is built in advance, and the problem turns into looking up morphologically analyzed eojeol in the PED.…”

Section: Korean Morphological Analysis and Word Sense Disambiguamentioning

confidence: 99%

Korean-Vietnamese Neural Machine Translation System With Korean Morphological Analysis and Word Sense Disambiguation

Nguyen

Shin

et al. 2019

IEEE Access

View full text Add to dashboard Cite

Although deep neural networks have recently led to great achievements in machine translation (MT), various challenges are still encountered during the development of Korean-Vietnamese MT systems. Because Korean is a morphologically rich language and Vietnamese is an analytic language, neither have clear word boundaries. The high rate of homographs in Korean causes word ambiguities, which causes problems in neural MT (NMT). In addition, as a low-resource language pair, there is no freely available, adequate Korean-Vietnamese parallel corpus that can be used to train translation models. In this paper, we manually established a lexical semantic network for the special characteristics of Korean as a knowledge base that was used for developing our Korean morphological analysis and word-sense disambiguation system: UTagger. We also constructed a large Korean-Vietnamese parallel corpus, in which we applied the state-of-the-art Vietnamese word segmentation method RDRsegmenter to Vietnamese texts and UTagger to Korean texts. Finally, we built a bi-directional Korean-Vietnamese NMT system based on the attentionbased encoder-decoder architecture. The experimental results indicated that UTagger and RDRsegmenter could significantly improve the performance of the Korean-Vietnamese NMT system, achieving remarkable results by 27.79 BLEU points and 58.77 TER points in Korean-to-Vietnamese direction and 25.44 BLEU points and 58.72 TER points in the reverse direction. INDEX TERMS Korean-Vietnamese machine translation, Korean-Vietnamese parallel corpus, lexical semantic network, morphological analysis, neural machine translation, word sense disambiguation. I. INTRODUCTION Neural machine translation based on the attention-based encoder-decoder model [1], [2] has emerged as the dominant paradigm in MT. It has achieved state-of-the-art performance in the translation of language pairs that have large amounts of training parallel corpora, such as English-French [3] and English-German [4]. However, it has shown poor translation quality in low-resource language pairs where training parallel corpora are scarce [5], [6]. Korean-Vietnamese is a low-resource language pair, and Korean-Vietnamese MT systems need to be built to serve The associate editor coordinating the review of this manuscript and approving it for publication was Yang Zhen.

show abstract