Automatic construction of generic Hausa language stop words list using term frequency-inverse document frequency

Abstract

The Hausa language, spoken by a large population, is considered a low-resource language in the field of Natural Language Processing (NLP), presenting unique challenges. Despite increasing efforts to address these challenges, the quality of existing resources, particularly datasets, remains uncertain. A critical task like stop word identification is often hindered by the absence of standardized resources. This study bridges this gap by leveraging the Term Frequency-Inverse Document Frequency (TF-IDF) approach alongside manual evaluation to develop a comprehensive stop word list for Hausa. Using datasets from four reputable online Hausa news sources, comprising 4,501 articles and 1,202,822 tokens, we applied TF-IDF with a threshold of 0.001 to each dataset, identifying 91 candidate stop words by intersecting results across the datasets. After manual examination, the list was narrowed to 76 final stop words. Compared to prior study, our list increased the number of identified stop words by 6%. This standardized resource advances Hausa NLP by facilitating more effective text processing tasks, such as sentiment analysis and machine translation, and lays the groundwork for further research in low-resource languages.