A Study on Term Weighting for Text Categorization: A Novel Supervised Variant of tf.idf

Domeniconi, Giacomo; Moro, Gianluca; Pasolini, Roberto; Sartori, Claudio

doi:10.5220/0005511900260037

Cited by 35 publications

(34 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where c(t,d) indicates the occurrence of term t appears in document d, and the denominator P i cðt i ; dÞ indicates the total number of terms in document d, D is the total number of documents in the dataset, and d t is the number of the documents a term t appeared in. Many researchers [26][27][28]36] tries to improve the performance of TF-IDF by proposing some modification to the original equation of TF-IDF. In the case of Bloom's taxonomy verb plays an important role to determine the level of the question.…”

Section: Modified Tf-idf (Term Weighting Methods Tfpos-idf)mentioning

confidence: 99%

“…It assigns scores to the importance of a word in a document, based on the lexical and morphological properties of the text. It is extensively used in many studies [25][26][27][28][29], and in question classification [3,6,20,23,[30][31][32] However, some researchers used TF-IDF as it is, while others proposed some enhancement to the way TF-IDF is calculated in order to improve it. This is because classical TF-IDF does not capture some useful information such as the impact of word distribution between various classes [6].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Question classification based on Bloom’s taxonomy cognitive domain using modified TF-IDF and word2vec

2020

View full text Add to dashboard Cite

The assessment of examination questions is crucial in educational institutes since examination is one of the most common methods to evaluate students' achievement in specific course. Therefore, there is a crucial need to construct a balanced and high-quality exam, which satisfies different cognitive levels. Thus, many lecturers rely on Bloom's taxonomy cognitive domain, which is a popular framework developed for the purpose of assessing students' intellectual abilities and skills. Several works have been proposed to automatically handle the classification of questions in accordance with Bloom's taxonomy. Most of these works classify questions according to specific domain. As a result, there is a lack of technique of classifying questions that belong to the multi-domain areas. The aim of this paper is to present a classification model to classify exam questions based on Bloom's taxonomy that belong to several areas. This study proposes a method for classifying questions automatically, by extracting two features, TFPOS-IDF and word2vec. The purpose of the first feature was to calculate the term frequency-inverse document frequency based on part of speech, in order to assign a suitable weight for essential words in the question. The second feature, pre-trained word2vec, was used to boost the classification process. Then, the combination of these features was fed into three different classifiers; K-Nearest Neighbour, Logistic Regression, and Support Vector Machine, in order to classify the questions. The experiments used two datasets. The first dataset contained 141 questions, while the other dataset contained 600 questions. The classification result for the first dataset achieved an average of 71.1%, 82.3% and 83.7% weighted F1-measure respectively. The classification result for the second dataset achieved an average of 85.4%, 89.4% and 89.7% weighted F1-measure respectively. The finding from this study showed that the proposed method is significant in classifying questions from multiple domains based on Bloom's taxonomy.

show abstract

Section: Modified Tf-idf (Term Weighting Methods Tfpos-idf)mentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Question classification based on Bloom’s taxonomy cognitive domain using modified TF-IDF and word2vec

2020

View full text Add to dashboard Cite

show abstract

“…According to this scheme, the higher the concentration of high-frequency terms in the positive category than in the negative one, the greater the contribution to classification. Domeniconi et al (2015) propose a supervised variant of IDF called inverse document frequency excluding category (IDFEC). Similar to IDF, IDFEC penalizes frequent terms, but different form IDF it avoids penalizing those terms occurring in several documents belonging to the same class.…”

Section: Background and Related Workmentioning

confidence: 99%

“…Similar to IDF, IDFEC penalizes frequent terms, but different form IDF it avoids penalizing those terms occurring in several documents belonging to the same class. Another variant also proposed in [7] results from combining IDFEC and RF, resulting in the IDFEC B scheme. Table 1 shows the definitions of the main scores presented above using the following notation [7,14]:…”

Section: Background and Related Workmentioning

confidence: 99%

A Flexible Supervised Term-Weighting Technique and its Application to Variable Extraction and Information Retrieval

Maisonnave

Delbianco²,

Tohmé³

et al. 2019

View full text Add to dashboard Cite

Successful modeling and prediction depend on effective methods for the extraction of domain-relevant variables. This paper proposes a methodology for identifying domain-specific terms. The proposed methodology relies on a collection of documents labeled as relevant or irrelevant to the domain under analysis. Based on the labeled document collection, we propose a supervised technique that weights terms based on their descriptive and discriminating power. Finally, the descriptive and discriminating values are combined into a general measure that, through the use of an adjustable parameter, allows to independently favor different aspects of retrieval such as maximizing precision or recall, or achieving a balance between both of them. The proposed technique is applied to the economic domain and is empirically evaluated through a human-subject experiment involving experts and non-experts in Economy. It is also evaluated as a term-weighting technique for query-term selection showing promising results. We finally illustrate the applicability of the proposed technique to address diverse problems such as building prediction models, supporting knowledge modeling, and achieving total recall.

show abstract

“…It would be difficult to find similar links due to that source units (e.g., opinions) and mapped units (e.g., fact of events) of links come from different domains (types of text). E.g., considering the following equation used for connecting two links (s, o), is the cosine similarity between the bag-of-words model of s using tf-idf weighting [30] and that model of s 2 , 0 ≤ a, b ≤ 1 are weights, θ is the threshold. It would be hard to compute weights a and b for finding similar links because we do not know the correlation between domain S and domain O.…”

Section: Composition For Implicit Referencesmentioning

confidence: 99%

Event Identification for Explicit and Implicit References on Microblogs

Katô

Yamamoto

et al. 2017

Journal of Information Processing

View full text Add to dashboard Cite

We address the problem of event identification on microblogs with special attention to implicit reference cases in which events are not referred to by event's information. Most studies identify events referred to by event's information, while there are many implicitly referred events by microblogs, which are difficult to identify for short text such as microblogs. We therefore tackled implicit reference cases by analyzing links from microblogs. The links are able to connect opinions or feeling to their referred events. The analysis of links is particularly important for certain types of implicit references. In addition, we predict reference type of a microblog for accurately ranking referred events. The experimental results suggest that our method was effective for implicit references and predicting reference type was essential for identifying implicitly or explicitly referred events together.

show abstract

A Study on Term Weighting for Text Categorization: A Novel Supervised Variant of tf.idf

Cited by 35 publications

References 18 publications

Question classification based on Bloom’s taxonomy cognitive domain using modified TF-IDF and word2vec

Question classification based on Bloom’s taxonomy cognitive domain using modified TF-IDF and word2vec

A Flexible Supervised Term-Weighting Technique and its Application to Variable Extraction and Information Retrieval

Event Identification for Explicit and Implicit References on Microblogs

Contact Info

Product

Resources

About