Application of Lemmatization and Summarization Methods in Topic Identification Module for Large Scale Language Modeling Data Filtering

Abstract

Abstract. The paper presents experiments with the topic identification module which is a part of a complex system for acquisition and storing large volumes of text data. The topic identification module processes each acquired data item and assigns it topics from a defined topic hierarchy. The topic hierarchy is quite extensive -it contains about 450 topics and topic categories. It can easily happen that for some narrowly focused topic there is not enough data for the topic identification training. Lemmatization is shown to improve the results when dealing with sparse data in the area of information retrieval, therefore the effects of lemmatization on topic identification results is studied in the paper. On the other hand, since the system is used for processing large amounts of data, a summarization method was implemented and the effect of using only the summary of an article on the topic identification accuracy is studied.

Dynamic Threshold Selection Method for Multi-label Newspaper Topic Identification

“…For the experiments the smaller collection containing the articles from the news serveř CeskéNoviny.cz separated from the whole corpus was used [17]. The collection contains 31 419 articles, divided into 27 000 training and 4 419 testing articles.…”

“…Lemmatization has been shown to improve the results when dealing with sparse data in the area of information retrieval [15] and spoken term detection [16] in highly inflected languages, on that account the experiments on the effects of lemmatization in the field of topic identification was performed [17]. As a result of these experiments the automatic text lemmatization is also applied in our work.…”

Do Non-State Actors Influence Climate Change Policy? Evidence from the Brazilian Nationally Determined Contributions for COP21

“…Although other methods could be used (i.e. Cosine Similarity, see Rezvani and Hashemi, 2012), when we employ pre-processing techniques such as lemmatisation, the text data is normalised, and the Jaccard Index efficiently captures similarities of concepts (Skorkovská, 2012).…”

“…The text was extracted from the documents and statements and pre-processed to standardise the words to enhance the effectiveness of the analysis. The pre-processing techniques were employed using natural language processing procedures: remove stopwords (connectives); tokenisation , which separates each word from the text; and, most important, lemmatisation , which transforms each inflected form of the words to its lemma, so words like “changing” or “changes” are transformed into “change,” and therefore the Jaccard index can capture more clearly the concepts expressed in the words (Skorkovská, 2012).…”

Score Normalization Methods Applied to Topic Identification

“…For the experiments a smaller collection containing 31 419 articles from the news serveř CeskéNoviny.cz separated from the whole corpus was used [13]. The collection contains articles published in the year 2011(January to October) and is divided into 27 000 training and 4 419 testing articles.…”