Tibetan vowel analysis with a multi-modal Mandarin-Tibetan speech corpus

Lobsang, Gyaltsen; Lu, Wenhuan; Honda, Kiyoshi; Wei, Jianguo; Guan, Wendan; Fang, Qiang; Dang, Jianwu

doi:10.1109/apsipa.2016.7820776

Cited by 2 publications

(2 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, Tibetan is rich in the phenomenon of linguistic partitions; for example, in the dictionary samples listed in the above literature, there are several noun-verb partitions such as "ུ་", "ིད་", and "ོར" [10]. The text does not propose any corresponding disambiguation strategy or method, which leads to the fact that when splitting a sentence; the noun-verb partitions will be taken for the verb of the sentence end, resulting in the splitting of many incorrect sentences [11]. We now provide an example of a sentence ending in "g-" without "shad" and discuss other uses of "shad".…”

Section: Introductionmentioning

confidence: 99%

Tibetan Sentence Boundaries Automatic Disambiguation Based on Bidirectional Encoder Representations from Transformers on Byte Pair Encoding Word Cutting Method

Li,

Zhao,

Wang

et al. 2024

Applied Sciences

View full text Add to dashboard Cite

Sentence Boundary Disambiguation (SBD) is crucial for building datasets for tasks such as machine translation, syntactic analysis, and semantic analysis. Currently, most automatic sentence segmentation in Tibetan adopts the methods of rule-based and statistical learning, as well as the combination of the two, which have high requirements on the corpus and the linguistic foundation of the researchers and are more costly to annotate manually. In this study, we explore Tibetan SBD using deep learning technology. Initially, we analyze Tibetan characteristics and various subword techniques, selecting Byte Pair Encoding (BPE) and Sentencepiece (SP) for text segmentation and training the Bidirectional Encoder Representations from Transformers (BERT) pre-trained language model. Secondly, we studied the Tibetan SBD based on different BERT pre-trained language models, which mainly learns the ambiguity of the shad (“།”) in different positions in modern Tibetan texts and determines through the model whether the shad (“།”) in the texts has the function of segmenting sentences. Meanwhile, this study introduces four models, BERT-CNN, BERT-RNN, BERT-RCNN, and BERT-DPCNN, based on the BERT model for performance comparison. Finally, to verify the performance of the pre-trained language models on the SBD task, this study conducts SBD experiments on both the publicly available Tibetan pre-trained language model TiBERT and the multilingual pre-trained language model (Multi-BERT). The experimental results show that the F1 score of the BERT (BPE) model trained in this study reaches 95.32% on 465,669 Tibetan sentences, nearly five percentage points higher than BERT (SP) and Multi-BERT. The SBD method based on pre-trained language models in this study lays the foundation for establishing datasets for the later tasks of Tibetan pre-training, summary extraction, and machine translation.

show abstract

Section: Introductionmentioning

confidence: 99%

Tibetan Sentence Boundaries Automatic Disambiguation Based on Bidirectional Encoder Representations from Transformers on Byte Pair Encoding Word Cutting Method

Li,

Zhao,

Wang

et al. 2024

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…For ancient Tibetan, a double shad represents the end of a paragraph in a poem or prose, and it functions as a period, question mark, semicolon, exclamation point, or ellipsis. A shad behind words or phrases indicates a period or comma [17] . With the development of Tibetan culture, people gradually simplified a double shad into a shad.…”

Section: Introductionmentioning

confidence: 99%

A Tibetan Sentence Boundary Disambiguation Model Considering the Components on Information on Both Sides of Shad

Gao

et al. 2023

Tsinghua Sci. Technol.

View full text Add to dashboard Cite

Sentence Boundary Disambiguation (SBD) is a preprocessing step for natural language processing.Segmenting text into sentences is essential for Deep Learning (DL) and pretraining language models. Tibetan punctuation marks may involve ambiguity about the sentences' beginnings and endings. Hence, the ambiguous punctuation marks must be distinguished, and the sentence structure must be correctly encoded in language models. This study proposed a component-level Tibetan SBD approach based on the DL model. The models can reduce the error amplification caused by word segmentation and part-of-speech tagging. Although most SBD methods have only considered text on the left side of punctuation marks, this study considers the text on both sides. In this study, 465 669 Tibetan sentences are adopted, and a Bidirectional Long Short-Term Memory (Bi-LSTM) model is used to perform SBD. The experimental results show that the F1-score of the Bi-LSTM model reached 96%, the most efficient among the six models. Experiments are performed on low-resource languages such as Turkish and Romanian, and high-resource languages such as English and German, to verify the models' generalization.

show abstract

Tibetan vowel analysis with a multi-modal Mandarin-Tibetan speech corpus

Cited by 2 publications

References 2 publications

Tibetan Sentence Boundaries Automatic Disambiguation Based on Bidirectional Encoder Representations from Transformers on Byte Pair Encoding Word Cutting Method

Tibetan Sentence Boundaries Automatic Disambiguation Based on Bidirectional Encoder Representations from Transformers on Byte Pair Encoding Word Cutting Method

A Tibetan Sentence Boundary Disambiguation Model Considering the Components on Information on Both Sides of Shad

Contact Info

Product

Resources

About