Syllable-based Neural Thai Word Segmentation

Chormai, Pattarawat; Prasertsom, Ponrawee; Cheevaprawatdomrong, Jin; Rutherford, Attapol T.

doi:10.18653/v1/2020.coling-main.407

Cited by 17 publications

(15 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Competitive Methods. We evaluate our proposed solution against two state-of-the-art methods namely DeepCut (DC) (Kittinaradorn et al, 2019) and AttaCut (AC) (Chormai et al, 2020). These methods are based on the Convolutional Neural Network (CNN) and trained on a generic corpus (BEST2009 (Boriboon et al, 2009)).…”

Section: Methodsmentioning

confidence: 99%

“…Evaluation Metrics. We use F1 score as the evaluation metric for the TWS task at character and word levels to avoid the overestimation of TWS (Chormai et al, 2020;Limkonchotiwat et al, 2020). Parameter Settings.…”

Section: Methodsmentioning

confidence: 99%

“…Word segmentation for low-resource languages is a challenging task due to the data limitation. Most Thai WS models report below 90% accuracy in domain-adaptation settings (Kittinaradorn et al, 2019;Chormai et al, 2020). Many researchers proposed data augmentation methods for Asian languages to increase the performance of WS models by using existing models' output as input to new models such as synthetic data, entropy parser, and character embedding (Zheng et al, 2018;Wang et al, 2019;Fung et al, 2004).…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Handling Cross- and Out-of-Domain Samples in Thai Word Segmentation

Limkonchotiwat¹,

Phatthiyaphaibun²,

Sarwar³

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

While word segmentation is a solved problem in many languages, it is still a challenge in continuous-script or low-resource languages. Like other NLP tasks, word segmentation is domain-dependent, which can be a challenge in low-resource languages like Thai and Urdu since there can be domains with insufficient data. This investigation proposes a new solution to adapt an existing domaingeneric model to a target domain, as well as a data augmentation technique to combat the low-resource problems. In addition to domain adaptation, we also propose a framework to handle out-of-domain inputs using an ensemble of domain-specific models called Multi-Domain Ensemble (MDE). To assess the effectiveness of the proposed solutions, we conducted extensive experiments on domain adaptation and out-of-domain scenarios. Moreover, we also proposed a multiple task dataset for Thai text processing, including word segmentation. For domain adaptation, we compared our solution to the state-of-the-art Thai word segmentation (TWS) method and obtained improvements from 93.47% to 98.48% at the character level and 84.03% to 96.75% at the word level. For out-of-domain scenarios, our MDE method significantly outperformed the state-of-the-art TWS and multi-criteria methods. Furthermore, to demonstrate our method's generalizability, we also applied our MDE framework to other languages, namely Chinese, Japanese, and Urdu, and obtained improvements similar to Thai's.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Handling Cross- and Out-of-Domain Samples in Thai Word Segmentation

Limkonchotiwat¹,

Phatthiyaphaibun²,

Sarwar³

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

show abstract

“…We train the χ 2 , t and WPE-based tokenizers for each language on Wikipedia articles for that For Thai and Chinese, we use the entire Wikipedia database, but for English we use the filtered Wiki103 dataset (Merity et al, 2016). English, Thai, and Chinese documents are tokenized with NLTK (Bird, 2006), Attacut (Chormai et al, 2020), and Stanford Word Segmenter (Tseng et al, 2005) respectively. We follow the same preprecessing steps for the training and the test documents: lemmatize and lowercase in English, and remove stopwords, symbols and digits for all languages.…”

Section: Methodsmentioning

confidence: 99%

More Than Words: Collocation Tokenization for Latent Dirichlet Allocation Models

Cheevaprawatdomrong,

Schofield,

Rutherford

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Traditionally, Latent Dirichlet Allocation (LDA) ingests words in a collection of documents to discover their latent topics using word-document co-occurrences. However, it is unclear how to achieve the best results for languages without marked word boundaries such as Chinese and Thai. Here, we explore the use of Pearson's chi-squared (χ 2 ) test, tstatistics, and Word Pair Encoding (WPE) to produce tokens as input to the LDA model. The χ 2 , t and WPE tokenizers are trained on Wikipedia text to look for words that should be grouped together, such as compound nouns, proper nouns, and complex event verbs. We propose a new metric for measuring the clustering quality in settings where the vocabularies of the models differ. Based on this metric and other established metrics, we show that topics trained with merged tokens result in topic keys that are clearer, more coherent, and more effective at distinguishing topics than those unmerged models.

show abstract

“…Aroonmanakun (2002) shows that syllable segmentation can resolve many word-level ambiguities in Thai. Plus, automatic syllable segmentation can be done at a near-perfect accuracy because the task is mostly solved by orthographic rules, assuming few typos exist in the data (Chormai et al, 2020). With syllable boundary indicators, we can avoid errors from word segmentation.…”

Section: Data Collection Proceduresmentioning

confidence: 99%

Thai Nested Named Entity Recognition Corpus

Buaphet¹,

Udomcharoenchaikit²,

Limkonchotiwat³

et al. 2022

Findings of the Association for Computational Linguistics: ACL 2022

Self Cite

View full text Add to dashboard Cite

This paper presents the first Thai Nested Named Entity Recognition (N-NER) dataset. Thai N-NER consists of 264,798 mentions, 104 classes, and a maximum depth of 8 layers obtained from news articles and restaurant reviews, a total of 4894 documents. Our work, to the best of our knowledge, presents the largest non-English N-NER dataset and the first non-English one with fine-grained classes. To understand the new challenges our proposed dataset brings to the field, we conduct an experimental study on (i) cutting edge N-NER models with the state-of-the-art accuracy in English and (ii) baseline methods based on wellknown language model architectures. From the experimental results, we obtain two key findings. First, all models produce poor F1 scores in the tail region of the class distribution. There is little or no performance improvement provided by these models with respect to the baseline methods with our Thai dataset. These findings suggest that further investigation is required to make a multilingual N-NER solution that works well across different languages.

show abstract

Syllable-based Neural Thai Word Segmentation

Cited by 17 publications

References 11 publications

Handling Cross- and Out-of-Domain Samples in Thai Word Segmentation

Handling Cross- and Out-of-Domain Samples in Thai Word Segmentation

More Than Words: Collocation Tokenization for Latent Dirichlet Allocation Models

Thai Nested Named Entity Recognition Corpus

Contact Info

Product

Resources

About