Die ontwikkeling van ’n woordafbreker en kompositumanaliseerder vir Afrikaans

Abstract

The development of a hyphenator and compound analyser for Afrikaans The development of two core-technologies for Afrikaans, viz. a hyphenator and a compound analyser is described in this article. As no annotated Afrikaans data existed prior to this project to serve as training data for a machine learning classifier, the core-technologies in question are first developed using a rule-based approach. The rule-based hyphenator and compound analyser are evaluated and the hyphenator obtains an fscore of 90,84%, while the compound analyser only reaches an f-score of 78,20%. Since these results are somewhat disappointing and/or insufficient for practical implementation, it was decided that a machine learning technique (memory-based learning) will be used instead. Training data for each of the two core-technologies is then developed using “TurboAnnotate”, an interface designed to improve the accuracy and speed of manual annotation. The hyphenator developed using machine learning has been trained with 39 943 words and reaches an fscore of 98,11% while the f-score of the compound analyser is 90,57% after being trained with 77 589 annotated words. It is concluded that machine learning (specifically memory-based learning) seems an appropriate approach for developing coretechnologies for Afrikaans.

Viability of Neural Networks for Core Technologies for Resource-Scarce Languages

“…The input is a sequence of characters and the target output is a sequence of characters interpolated with compound boundaries (+) and valence morpheme boundaries (_) as seen in Table 3. This format is similar to that described in [22], with the exception that the sequence is space separated and is appended with an end-of-sequence (.) marker, which preliminary experiments indicated improved the accuracy of sequence length prediction.…”

“…For compound analysis, results are compared with those reported in [22] as a baseline, which were achieved using a memory-based learner. They are additionally compared with those achieved by CatBoost, a gradient-boosted decision tree (GBDT) implementation [68], which was found to be the best-performing machine learning algorithm of those evaluated by the authors (machine learning methods tested included kNN's (IBk), non-boosted decision trees, Random Forest, SVM's, One Rule, and Naïve Bayes).…”

“…For isiXhosa, a machine learning based lemmatiser was developed in 2015 [20] using data annotated by a rule-based lemmatiser, developed in 2014 [2]. A rule-based compound analyser was developed in 2004 [21], followed by a machine learning based method in 2008 [22]. Several other machine learning algorithms are currently in use in core technologies for SA languages.…”

“…Decision trees were used for automatic compound analysis of Afrikaans in [21]. Later work on compound analysis used TiMBL, a memory-based learner which employs the k-Nearest Neighbor (k-NN) algorithm [22]. The k-NN algorithm was also used to develop the Lemma Identifier for Afrikaans (Lia) [19,26].…”

Proceedings of the First Workshop on Computational Approaches to Compound Analysis (ComAComA 2014)

“…The manual is based on the annotation guidelines that were developed during the CKarma project (CTexT, 2005;Pilon et al, 2008). These initial guidelines only apply to Afrikaans, and was hence extended to handle Dutch compounds as well as more complicated cases not foreseen in the original CKarma guidelines.…”