Automatic Multilingual Stopwords Identification from Very Small Corpora

Abstract

Tools for Natural Language Processing work using linguistic resources, that are languagespecific. The complexity of building such resources causes many languages to lack them. So, learning them automatically from sample texts would be a desirable solution. This usually requires huge training corpora, which are not available for many local languages and jargons, lacking a wide literature. This paper focuses on stopwords, i.e., terms in a text which do not contribute in conveying its topic or content. It provides two main, inter-related and complementary, methodological contributions: (i) it proposes a novel approach based on term and document frequency to rank candidate stopwords, that works also on very small corpora (even single documents); and (ii) it proposes an automatic cutoff strategy to select the best candidates in the ranking, thus addressing one of the most critical problems in the stopword identification practice. Nice features of these approaches are that (i) they are generic and applicable to different languages, (ii) they are fully automatic, and (iii) they do not require any previous linguistic knowledge. Extensive experiments show that both are extremely effective and reliable. The former outperforms all comparable approaches in the state-of-the-art, both in terms of performance (Precision stays at 100% or nearly so for a large portion of the top-ranked candidate stopwords, while Recall is quite close to the maximum reachable in theory.) and in smooth behavior (Precision is monotonically decreasing, and Recall is monotonically increasing, allowing the experimenter to choose the preferred balance.). The latter is more flexible than existing solutions in the literature, requiring just one parameter intuitively related to the balance between Precision and Recall one wishes to obtain.

Complement Recognition-Based Formal Concept Analysis for Automatic Extraction of Interpretable Concept Taxonomies from Text

“…The resources learned by BLA-BLA may be used as returned by the system, and/or be taken as a basis for further manual refinements. It currently includes several techniques that allow us to learn in a fully automatic way linguistic resources for language identification [26], stopword removal [27], term normalization [28], syntax checking [29] and concept taxonomies [30]. Whenever more texts become available for the language, it is easy to run BLA-BLA again and obtain updated resources.…”

Text Information Retrieval in Tetun

“…For the text processing tasks, Jurish et al [16] proposed a Hidden Markov Model-based approach to segment automatically text documents into tokens and sentences. Ferilli [17] presented a term-document frequency approach that automatically detects stopwords from a small amount of the corpora and stated that it was the most effective approach which outperformed the classic term frequency [18,2,19] and the normalized inverse document frequency of Lo et al [20]. Trishala and Mamatha [21] proposed a rule-based Kannada stemmer relying on an unsupervised approach using k-means algorithm, and Thangarasu and Inbarani [22] presented an analogy removal stemmer that automatically stem Tamil words from the text corpora.…”