Sentence Boundary Detection in Turkish News with Regular Expressions

Ozbey, Can; Dincsoy, Ozge

doi:10.1109/siu.2019.8806556

Cited by 3 publications

(1 citation statement)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The data crawled from the Internet for low-resource languages may vary from an article or a paragraph. However, too long sequences will increase the training cost, so the text must be segmented into sentences [11] . Segmenting text into sentences is a simple task for humans, but it is challenging for computers to learn sentence-ending features.…”

Section: Introductionmentioning

confidence: 99%

A Tibetan Sentence Boundary Disambiguation Model Considering the Components on Information on Both Sides of Shad

Gao

et al. 2023

Tsinghua Sci. Technol.

View full text Add to dashboard Cite

Sentence Boundary Disambiguation (SBD) is a preprocessing step for natural language processing.Segmenting text into sentences is essential for Deep Learning (DL) and pretraining language models. Tibetan punctuation marks may involve ambiguity about the sentences' beginnings and endings. Hence, the ambiguous punctuation marks must be distinguished, and the sentence structure must be correctly encoded in language models. This study proposed a component-level Tibetan SBD approach based on the DL model. The models can reduce the error amplification caused by word segmentation and part-of-speech tagging. Although most SBD methods have only considered text on the left side of punctuation marks, this study considers the text on both sides. In this study, 465 669 Tibetan sentences are adopted, and a Bidirectional Long Short-Term Memory (Bi-LSTM) model is used to perform SBD. The experimental results show that the F1-score of the Bi-LSTM model reached 96%, the most efficient among the six models. Experiments are performed on low-resource languages such as Turkish and Romanian, and high-resource languages such as English and German, to verify the models' generalization.

show abstract

Section: Introductionmentioning

confidence: 99%