Selection of the most relevant terms based on a max-min ratio metric for text classification

Rehman, Abdur; Javed, Kashif; Babri, Haroon A.; Asim, Muhammad Nabeel

doi:10.1016/j.eswa.2018.07.028

Cited by 43 publications

(35 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Uysal proposes an improved global feature selection scheme (IGFSS) that marks features based on their ability to distinguish between classes and uses these marks when generating feature sets [29]. In 2018, Rehman proposed the MMR (Max-Min Ratio) algorithm, which performs feature selection based on the product of max-min ratio of the true positives and false positives and their difference [30].…”

Section: A Overview Of Feature Selection Methods For Text Classificamentioning

confidence: 99%

A Novel Feature Selection Method Based on Category Distribution Ratio in Text Classification

Zong¹,

Bian²

2019

Proceedings of the 2019 International Conference on Mathematics, Big Data Analysis and Simulation and Modelling (MBDASM 2019)

View full text Add to dashboard Cite

In text classification, texts are represented as a high-dimensional and sparse matrix, whose dimension is the same as the total number of terms of all texts. Using all terms for text classification tasks will affect the accuracy and efficiency. Feature selection algorithm can select some features most relevant to text category and reduce the dimension of text representation vector. In this paper, we propose a new feature ranking metric as category distribution ratio (CDR) which takes the true positive rate and false positive rate and their difference of a term into account while estimating the significance of a term. To prove the effectiveness of the proposed feature selection algorithm, we compare its performance against six metrics (balanced accuracy measure (ACC), odds ratio (OR), Gini index (GI), max-min Ratio (MMR), normalized difference measure(NDM)，chi-square (CHI)) on three benchmark data sets (20newsgropus, Ohsumed, Reuters 21578) using multinomial naive Bayes, support vector machines and k-nearest neighbor classifiers. The experimental results show that the classification evaluation index macro F1 based on CDR feature selection is higher than the other six algorithms.

show abstract

Section: A Overview Of Feature Selection Methods For Text Classificamentioning

confidence: 99%

A Novel Feature Selection Method Based on Category Distribution Ratio in Text Classification

Zong¹,

Bian²

2019

Proceedings of the 2019 International Conference on Mathematics, Big Data Analysis and Simulation and Modelling (MBDASM 2019)

View full text Add to dashboard Cite

show abstract

“…In Micro‐ F ‐measure, first precision, and recall are calculated globally, and then F ‐measure is calculated with these new values [48 ]

Precisio {normaln}^{μ} = \sum_{k} precisio {normaln}_{k}

Recal {normall}^{μ} = \sum_{k} recal {normall}_{k}

where μ indicates micro‐averaging.…”

Section: Empirical Evaluationmentioning

confidence: 99%

“…Equation (8 ) provides the expressions for Micro F ‐measure [48 ].

F normal_measuer^{μ} = \frac{2 \times Precisio {normaln}^{μ} \times Recal {normall}^{μ}}{Precisio {normaln}^{μ} + Recal {normall}^{μ}}

The results of these two methods may be quite different.…”

Section: Empirical Evaluationmentioning

confidence: 99%

Automatic summarising of user stories in order to be reused in future similar projects

et al. 2020

View full text Add to dashboard Cite

User stories play an important role in agile development systems. In this study, a method of summarising user stories is proposed to reuse them in the future. To enhance the results, quality improvement should be made on user stories. It would help developers build better results, and it may also lead to omitting some essential information. To avoid such issues, user stories are duplicated in two exact similar groups, and quality improvement is made on one set while the other set remains unattained. With the help of a modified bag of words and a verb parser, a collection of keywords and key verbs are extracted for both groups. Afterwards, automatic user stories are made, and then an expert improves them. Next, some experts choose between the results and select the better ones. The result is evaluated by applying different experiments on the framework and prototype implementation on 14 data sets of a user story from industry and a fake data set from Duke University. The result showed 97% of micro F ‐measure and 93% of macro F ‐measure, which are promising. These new user stories can be used as the base user stories in future similar projects.

show abstract

“…Broadly, feature selection approaches can be divided into three classes namely wrapper, embedded, and filter [8], [9]. In recent years, researchers have proposed various filter based feature selection methods to raise the performance of document text classification [34].…”

Section: Related Workmentioning

confidence: 99%

A Robust Hybrid Approach for Textual Document Classification

Asim

Khan

Malik

et al. 2019

2019 International Conference on Document Analysis and Recognition (ICDAR)

Self Cite

View full text Add to dashboard Cite

Text document classification is an important task for diverse natural language processing based applications. Traditional machine learning approaches mainly focused on reducing dimensionality of textual data to perform classification. This although improved the overall classification accuracy, the classifiers still faced sparsity problem due to lack of better data representation techniques. Deep learning based text document classification, on the other hand, benefitted greatly from the invention of word embeddings that have solved the sparsity problem and researchers focus mainly remained on the development of deep architectures. Deeper architectures, however, learn some redundant features that limit the performance of deep learning based solutions. In this paper, we propose a two stage text document classification methodology which combines traditional feature engineering with automatic feature engineering (using deep learning). The proposed methodology comprises a filter based feature selection (FSE) algorithm followed by a deep convolutional neural network. This methodology is evaluated on the two most commonly used public datasets, i.e., 20 Newsgroups data and BBC news data. Evaluation results reveal that the proposed methodology outperforms the state-of-the-art of both the (traditional) machine learning and deep learning based text document classification methodologies with a significant margin of 7.7% on 20 Newsgroups and 6.6% on BBC news datasets.

show abstract

Selection of the most relevant terms based on a max-min ratio metric for text classification

Cited by 43 publications

References 30 publications

A Novel Feature Selection Method Based on Category Distribution Ratio in Text Classification

A Novel Feature Selection Method Based on Category Distribution Ratio in Text Classification

Automatic summarising of user stories in order to be reused in future similar projects

A Robust Hybrid Approach for Textual Document Classification

Contact Info

Product

Resources

About