Distributed Training for Multilingual Combined Tokenizer using Deep Learning Model and Simple Communication Protocol

Purwanto, Christian Nathaniel; Hermawan, Arya Tandy; Santoso, Joan; Gunawan, .

doi:10.1109/icoris.2019.8874898

Cited by 2 publications

(6 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on the experimental results in previous studies, the best performance of the Bi-LSTM proposed in this study provides the most significant increase of approximately 13% compared to other approaches that do not use Deep Learning. However, compared with the Bi-LSTM that has been proposed by [21], there was an increase of approximately 2%. The reason is that the results of the proposed approach are using two labels and while in [21] approach uses four labels.…”

Section: Resultsmentioning

confidence: 92%

“…However, compared with the Bi-LSTM that has been proposed by [21], there was an increase of approximately 2%. The reason is that the results of the proposed approach are using two labels and while in [21] approach uses four labels. The use of two labels can give the best results compared to 4 labels in previous studies, especially in sentence boundary detection research.…”

Section: Resultsmentioning

confidence: 92%

“…Several sequential tagging methods will be used as a comparison method in this proposed approach, namely Maximum Entropy, Decision Tree, and Naïve Bayes. In addition to using several traditional non-Deep Learning models, the performance of the proposed method is also compared with previous studies using Bi-LSTM by Purwanto et al [21]. The experimental results can be seen in Table 2.…”

Section: Resultsmentioning

confidence: 99%

“…Previous researchers have worked on several machine learning approaches on Sentence Boundary Detection, i.e., Unsupervised [14], Rule-Based Method [13][15], Maximum Entropy [16], Hidden Markov Model [17], Conditional Random Field [18], Support Vector Machine [19], and Confusion Networks [20]. We use a deep learning approach to detect the sentence boundary as in our previous work [21]. Sentence Boundary Detection has been studied on other languages like English [22], Portuguese [23], French [24], Vietnamese [25], Chinese [26], Japanese [27], Marathi [28], Kannada [29], Arabic [30], and Urdu [31].…”

Section: Introductionmentioning

confidence: 99%

“…Another study is in Thai with Bi-LSTM CNN Approach [32]. In Indonesian, Sentence Boundary Detection has been presented using Maximum Entropy [33] and Bidirectional LSTM [21].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Indonesian Sentence Boundary Detection using Deep Learning Approaches

et al. 2021

Self Cite

View full text Add to dashboard Cite

Detecting the sentence boundary is one of the crucial pre-processing steps in natural language processing. It can define the boundary of a sentence since the border between a sentence, and another sentence might be ambiguous. Because there are multiple separators and dynamic sentence patterns, using a full stop at the end of a sentence is sometimes inappropriate. This research uses a deep learning approach to split each sentence from an Indonesian news document. Hence, there is no need to define any handcrafted features or rules. In Part of Speech Tagging and Named Entity Recognition, we use sequence labeling to determine sentence boundaries. Two labels will be used, namely O as a non-boundary token and E as the last token marker in the sentence. To do this, we used the Bi-LSTM approach, which has been widely used in sequence labeling. We have proved that our approach works for Indonesian text using pre-trained embedding in Indonesian, as in previous studies. This study achieved an F1-Score value of 98.49 percent. When compared to previous studies, the achieved performance represents a significant increase in outcomes..

show abstract

Section: Resultsmentioning

confidence: 92%

Section: Resultsmentioning

confidence: 92%

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…Another study is in Thai with Bi-LSTM CNN Approach [32]. In Indonesian, Sentence Boundary Detection has been presented using Maximum Entropy [33] and Bidirectional LSTM [21].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Indonesian Sentence Boundary Detection using Deep Learning Approaches

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

An adaptable sentence segmentation based on Indonesian rules

Petrus¹,

Ermatita

Sukemi

et al. 2023

IJ-AI

View full text Add to dashboard Cite

<p>Sentence segmentation that breaks textual data strings into individual sentences is an important phase in natural language processing (NLP). Each word in the string that is added a punctuation mark such as a period, question mark, or exclamation point, becomes the location for splitting the string. Humans can easily see the punctuation and split the string into sentences, but not machines. Basically, the three punctuation marks also perform other functions so that the sentence segmentation process must really be able to detect whether a word marked with punctuation is a sentence boundary or not. This research proposes a sentence segmentation system called segmentasi kalimat bahasa Indonesia (SKBI) or Indonesian language Sentence Segmentation by applying a set of rules and can be used in Indonesian texts and can be adapted for English. There are 34 rules built with a combination of 27 fairly complete features that contribute to this research. The experimental results for the Indonesian text show that the SKBI is able to achieve an F1-Score of 96.89% and 97.07% for English. Both need to be improved but now better than previous research.</p>

show abstract

Distributed Training for Multilingual Combined Tokenizer using Deep Learning Model and Simple Communication Protocol

Cited by 2 publications

References 12 publications

Indonesian Sentence Boundary Detection using Deep Learning Approaches

Indonesian Sentence Boundary Detection using Deep Learning Approaches

An adaptable sentence segmentation based on Indonesian rules

Contact Info

Product

Resources

About