Analytical evaluation of term weighting schemes for text categorization

Altınçay, Hakan; Erenel, Zafer

doi:10.1016/j.patrec.2010.03.012

Cited by 51 publications

(33 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Mmutual Information method is used as a method for evaluating the importance of terms (Mmutual Information) [20]; 3) Classification of texts according to scientific areas. The method of k nearest neighbour (kNN) is used for text classification [ 20], [21]; Working with the text files in the corpus with the purpose to perform statistical calculations requires the following preliminary steps: 1) To pre-translate to .txt format the files of different formats (pdf, doc,. docx) in the corpus; 2) To delete all hyphenation beforehand; 3) To perform lemmatization of all the text files in the corpus, to delete all punctuation marks, to change all uppercase letters to lowercase letters.…”

Section: The Procedures Of the University's Information Resources mentioning

confidence: 99%

Information Models and Methods of the University's Scientific Knowledge Life Cycle Support

Zhomartkyzy¹,

Tatiana²,

Miłosz³

2014

Annals of Computer Science and Information Systems

View full text Add to dashboard Cite

Abstract-The main aim of this work is to develop methods and technologies of the university's scientific knowledge management. The paper examines the concept of knowledge management and life-cycle processes of the university's scientific knowledge. Text Mining and Semantic Web technologies are used to develop the ontological information model and for information resources processing. The paper describes the developed information model of the university's scientific knowledge, the methods of forming scientific profiles, and the concept of the university's scientific knowledge semantic portal. I. INTRODUCTIONK NOWLEDGE , intellectual property, and intellectual resources have been understood in recent decades as a major driving force of the economy of the "third wave" (the new economy), the economy based on knowledge of expanded reproduction [1]. Knowledge as intellectual capital is gradually becoming one of the most important factors in the development of economy and the society.For a modern institution of higher education as an open social and economic self-organizing system the processes of creation, accumulation, and dissemination of knowledge is becoming a key factor for training competitive specialists.Paraphrasing [2], we can determine that the university's scientific knowledge as the combination of data and information with added opinions, skills and experience of the university's scientists and faculty members is a valuable asset that can provide an advantage in the market of educational services in project activities, scientific and practical work, and innovative activities.The intellectual capital or intangible assets of the university are the source of new scientific knowledge. Edvinsson developed a hierarchical structure of intellectual capital of a higher education institution -a "Skandia Value Scheme" model [3]. The main components of this model are human assets (the capital) and innovating capital which are tacit knowledge.Tacit knowledge form the human capital which is embodied in the university staff as a body of knowledge, qualifications, each employee's innovation, as the system of values, culture and philosophy of the institution. It is believed that practical tacit knowledge is the key to decision making and management.There is a continuous exchange between explicit and implicit knowledge and their transformation. [4]. Nonaka I., Takeuchi H. suggested a cyclical process of knowledge transformation: socialization, externalization, combination, and internalization.Kantner defines the concept of knowledge management as a strategy for the organization and the process of knowledge transformation [5], [6]. Currently, there is the increasing number of research papers devoted to the issue of the community of practitioners' knowledge management and the implementation of individual processes of knowledge transformation. [7], [8].The European concept of knowledge management [2] identifies five processes of knowledge life cycle: knowledge identification, creation, storage, distribution and use. Knowledge ...

show abstract

Section: The Procedures Of the University's Information Resources mentioning

confidence: 99%

Information Models and Methods of the University's Scientific Knowledge Life Cycle Support

Zhomartkyzy¹,

Tatiana²,

Miłosz³

2014

Annals of Computer Science and Information Systems

View full text Add to dashboard Cite

show abstract

“…Bag of word is mainly used for document representation in each text categorization and information retrieval. An important part of the text categorization systems using the bag of word representation therefore known as term-weighting scheme that is responsible for deciding while relevant term is for describing the content of a document [28,29,30,31]. Term-weighting schemes be term frequency, where the value of a word in a document is given by its frequency of occurrence in the document.Even though, as a result of capturing statistical data from the original document provides simplicity of vector space model.…”

Section: Introductionmentioning

confidence: 99%

Evaluation of Clustering around Weighted Prototype and Genetic Algorithm for Document Categorization

Jain¹,

Shrivastava²

2015

IJCA

View full text Add to dashboard Cite

Document clustering is very important in the field of text categorization. Genetic algorithm, which is an optimization based technique which can be applied for finding out the best cluster centres easily by computing fitness values of data points. While clustering around weighted prototype technique is especially helpful when proper pairwise similarities are available. This technique does not find global solution of the objective function. Experimental result shows that F-measure and Normalized mutual information of genetic algorithm is better than clustering around weighted prototype for 20 Newsgroup dataset. F-measure and accuracy of genetic algorithm is better than clustering around weighted prototype for the Reuter-21578 dataset.

show abstract

“…In order to clarify the major differences among weighting schemes that lead to inconsistent performances on different categories, a common analysis framework is recently developed by the authors of this study [15]. It is shown that the term weights can be expressed as a function of two term specific parameters namely, the term occurrence probability in the category under concern (p + ) and in its complement (p − ) [15].…”

Section: Introductionmentioning

confidence: 99%

“…It is shown that the term weights can be expressed as a function of two term specific parameters namely, the term occurrence probability in the category under concern (p + ) and in its complement (p − ) [15]. The weight expressions are also presented in the form of contour lines that are curves along which the term weighting schemes provide constant values to illustrate the distribution of equally weighted terms.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Using the absolute difference of term occurrence probabilities in binary text categorization

Altınçay

Erenel

2010

Appl Intell

Self Cite

View full text Add to dashboard Cite

In this study, the differences among widely used weighting schemes are studied by means of ordering terms according to their discriminative abilities using a recently developed framework which expresses term weights in terms of the ratio and absolute difference of term occurrence probabilities. Having observed that the ordering of terms is dependent on the weighting scheme under concern, it is emphasized that this can be explained by the way different schemes use term occurrence differences in generating term weights. Then, it is proposed that the relevance frequency which is shown to provide the best scores on several datasets can be improved by taking into account the way absolute difference values are used in other widely used schemes. Experimental results on two different datasets have shown that improved F 1 scores can be achieved.

show abstract

Analytical evaluation of term weighting schemes for text categorization

Cited by 51 publications

References 19 publications

Information Models and Methods of the University's Scientific Knowledge Life Cycle Support

Information Models and Methods of the University's Scientific Knowledge Life Cycle Support

Evaluation of Clustering around Weighted Prototype and Genetic Algorithm for Document Categorization

Using the absolute difference of term occurrence probabilities in binary text categorization

Contact Info

Product

Resources

About