Towards an unsupervised morphological segmenter for isiXhosa

Mzamo, Lulamile; Helberg, A.S.J.; Bosch, Sonja E.

doi:10.1109/robomech.2019.8704816

Cited by 5 publications

(3 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also experimented with inserting a segment boundary based on whether the entropy increases between adjacent positions, as well as an objective that compares the sum of the left and right entropies to the mean over all the entropies in the word to perform relatively thresholding. These are similar to objective functions proposed by Mzamo et al (2019b). However, in our experiments the constant entropy objective performed substantially better than either of those approaches.…”

Section: Entropy-based Modelsupporting

confidence: 77%

Canonical and Surface Morphological Segmentation for Nguni Languages

Moeng¹,

Reay²,

Daniels³

et al. 2021

Preprint

View full text Add to dashboard Cite

Morphological Segmentation involves decomposing words into morphemes, the smallest meaning-bearing units of language. This is an important NLP task for morphologically-rich agglutinative languages such as the Southern African Nguni language group. In this paper, we investigate supervised and unsupervised models for two variants of morphological segmentation: canonical and surface segmentation. We train sequence-to-sequence models for canonical segmentation, where the underlying morphemes may not be equal to the surface form of the word, and Conditional Random Fields (CRF) for surface segmentation. Transformers outperform LSTMs with attention on canonical segmentation, obtaining an average F1 score of 72.5% across 4 languages. Feature-based CRFs outperform bidirectional LSTM-CRFs to obtain an average of 97.1% F1 on surface segmentation. In the unsupervised setting, an entropy-based approach using a character-level LSTM language model fails to outperforms a Morfessor baseline, while on some of the languages neither approach perform much better than a random baseline. We hope that the high performance of the supervised segmentation models will help to facilitate the development of better NLP tools for Nguni languages.

show abstract

Section: Entropy-based Modelsupporting

confidence: 77%

Canonical and Surface Morphological Segmentation for Nguni Languages

Moeng¹,

Reay²,

Daniels³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In an unsupervised segmentation approach, the algorithm models from raw texts to produce respective segments. For this kind of technique, Mzamo et al (2019) (2015) have leveraged this dataset to develop surface segmentation models. Among these, Cotterell et al ( 2015) stand out as they not only developed a system called CHIPMUNK but also rigorously evaluated its performance.…”

Section: Related Workmentioning

confidence: 99%

Generation of segmented isiZulu text

Mkhwanazi,

Marais

2024

JDHASA

View full text Add to dashboard Cite

The complex morphology, conjunctive orthography and widespread occurrence of morphophonological alternation in the Nguni languages have given rise to several efforts towards morphological segmentation of tokens of Nguni languages. For supervised methods, annotated data is required, which currently exists as canonically segmented data in the NCHLT corpus and surface segmented data in the Ukwabelana corpus. In this paper, we present a method and segmentation strategy based on a computational grammar for isiZulu. The grammar, which itself has some limitations in processing speed and robustness to unexpected input, is used to create a new set of segmentations for the tokens of the Ukwabelana corpus.By training various models with the same architecture but on different datasets, we first show that our approach enables us to match the performance of a model trained on pre-existing data. We also show that our approach provides the flexibility to determine a suitable segmentation strategy and to generate data that reflects this strategy.

show abstract

“…When widening the search for related literature further, we see that there exists a verb parser and generator (Pretorius et al (2017)), morphological analysers (Pretorius & Bosch (2009), ), morphological segmenters (e.g., Mzamo et al (2019), Moeng et al (2021)), language models (e.g., Myoya et al (2023)), and a Grammatical Framework (GF) grammar…”

Section: Related Workmentioning

confidence: 99%

Algorithm for Assisting Grammarians when Extracting Phonological Conditioning Rules for Nguni languages

Mahlaza,

Khumalo

2024

JDHASA

View full text Add to dashboard Cite

Text generation models, the core technology that underpins chatbots such as ChatGPT [1], that are created to support morphologically complex African languages require the modelling of subword processes such as phonological conditioning. Since we rely on explicit phonological conditioning rules that are manually identified by grammarians to determine the extent to which such models are able to perform for such languages, there is a need to assist grammarians via computational solutions to increase their coverage of known rules. At present, there are no existing algorithms to extract the rules for such processes and therefore enable the creation of building better text generation models. We present a new algorithm for extracting phonological conditioning rules for Nguni languages. All the rules extracted by the algorithm are valid when the input word and associated morphemes are judged to be valid. The algorithm has the potential to improve the productivity of grammarians and enable the creation of modern text generation technologies that support and promote under-resourced languages.

show abstract

Towards an unsupervised morphological segmenter for isiXhosa

Cited by 5 publications

References 19 publications

Canonical and Surface Morphological Segmentation for Nguni Languages

Canonical and Surface Morphological Segmentation for Nguni Languages

Generation of segmented isiZulu text

Algorithm for Assisting Grammarians when Extracting Phonological Conditioning Rules for Nguni languages

Contact Info

Product

Resources

About