Context-Driven Corpus-Based Model for Automatic Text Segmentation and Part of Speech Tagging in Setswana Using OpenNLP Tool

Dibitso, Mary Ambrossine; Owolawi, Pius A.; Ojo, Sunday O.

doi:10.1007/978-3-030-34974-5_6

Cited by 3 publications

(2 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…LiveLanguage and the UKC integrate both existing third-party resources and linguistic data collected through collaborations with universities. Examples of such collaborations include Mongolia (Batsuren et al, 2019), Scotland (Bella et al, 2020), India (Chandran Nair et al, 2022, Palestine (Khalilia et al, 2023), and South Africa (Dibitso et al, 2019). Striving to ensure that such collaborations are beneficial to local speaker communities and to avoid exploitative practices, LiveLanguage collaborations adopt a methodology based on co-creation and local empowerment, with the following characteristics: (a) representatives of local communities are leading the formulation of problems and needs, as well as the subsequent specifications of the language resources to be developed; (b) tools, infrastructure, and know-how are provided to local communities if needed, in order to embed solutions sustainably; (c) intellectual property rights stay with the local community; (d) language resources are integrated into the global LiveLanguage ecosystem, giving worldwide visibility to the results through the UKC database and the LiveLanguage data catalog.…”

Section: Addressing Epistemic Injustice In Language Technology: the L...mentioning

confidence: 99%

Diversity and language technology: how language modeling bias causes epistemic injustice

Helm,

Bella,

Koch

et al. 2024

Ethics Inf Technol

View full text Add to dashboard Cite

It is well known that AI-based language technology—large language models, machine translation systems, multilingual dictionaries, and corpora—is currently limited to three percent of the world’s most widely spoken, financially and politically backed languages. In response, recent efforts have sought to address the “digital language divide” by extending the reach of large language models to “underserved languages.” We show how some of these efforts tend to produce flawed solutions that adhere to a hard-wired representational preference for certain languages, which we call language modeling bias. Language modeling bias is a specific and under-studied form of linguistic bias were language technology by design favors certain languages, dialects, or sociolects with respect to others. We show that language modeling bias can result in systems that, while being precise regarding languages and cultures of dominant powers, are limited in the expression of socio-culturally relevant notions of other communities. We further argue that at the root of this problem lies a systematic tendency of technology developer communities to apply a simplistic understanding of diversity which does not do justice to the more profound differences that languages, and ultimately the communities that speak them, embody. Drawing on the concept of epistemic injustice, we point to the broader ethico-political implications and show how it can lead not only to a disregard for valuable aspects of diversity but also to an under-representation of the needs of marginalized language communities. Finally, we present an alternative socio-technical approach that is designed to tackle some of the analyzed problems.

show abstract

Section: Addressing Epistemic Injustice In Language Technology: the L...mentioning

confidence: 99%

Diversity and language technology: how language modeling bias causes epistemic injustice

Helm,

Bella,

Koch

et al. 2024

Ethics Inf Technol

View full text Add to dashboard Cite

show abstract

“…It is an agglutinative language where some words can be formed by adding suffixes and prefixes to the root word. (Dibitso et al,2019). As with any other language, the main problem of Setswana PoS tagging is the ambiguities that may appear at different levels.…”

Section: Introductionmentioning

confidence: 99%

An Hybrid Part of Speech Tagger for Setswana Language using a Voting Method

Dibitso¹,

Owolawi²,

Ojo³

2022

ICONIC

View full text Add to dashboard Cite

Part-of Speech (PoS) tagging is a corpus linguistics that deals with assigning appropriate lexical categories to each word in a sentence. To effectively address challenges associated with PoS tagging, several Natural Language Processing (NLP) tasks modelling techniques have been employed, including Conditional Random Fields (CRF), Support Vector Machines (SVM), and Decision Trees in diverse languages. These PoS taggers implement the process of associating the correct PoS (nouns, verbs, adjectives, adverbs, etc.) with each word in a sentence. However, creating language resources is an expensive process for many languages, including the indigenous languages of South Africa that are classified as resource-scarce. Therefore, using Setswana as a language with limited resources, this study explores and applies methods to increase the utilization of existing resources and tagger accuracy. This is done using Setswana's two PoS taggers: a Maximum Entropy (MaxEnt) and an SVM, which achieved an accurateness of 94.4 per cent and 95.59 per cent respectively. To find errors in the taggers, an error analysis is carried out. The Setswana PoS Tagger was then built using a voting algorithm to improve results and attain 97.06 per cent accuracy. The combination of taggers reduces the error rate by up to 2.01 per cent.

show abstract

Reactive-Optimized Sentence Detection In Kubernetes Using OpenNLP And Native GraalVM Image With Framework Metric Comparison

Sharma¹,

Tahiliani²,

Dubey

2023

2023 4th International Conference for Emerging Technology (INCET)

View full text Add to dashboard Cite

Context-Driven Corpus-Based Model for Automatic Text Segmentation and Part of Speech Tagging in Setswana Using OpenNLP Tool

Cited by 3 publications

References 6 publications

Diversity and language technology: how language modeling bias causes epistemic injustice

Diversity and language technology: how language modeling bias causes epistemic injustice

An Hybrid Part of Speech Tagger for Setswana Language using a Voting Method

Reactive-Optimized Sentence Detection In Kubernetes Using OpenNLP And Native GraalVM Image With Framework Metric Comparison

Contact Info

Product

Resources

About