A Hybrid Statistical Data Pre-processing Approach for Language-Independent Text Classification

Abstract

Abstract. Data pre-processing is an important topic in Text Classification (TC).It aims to convert the original textual data in a data-mining-ready structure, where the most significant text-features that serve to differentiate between textcategories are identified. Broadly speaking, textual data pre-processing techniques can be divided into three groups: (i) linguistic, (ii) statistical, and (iii) hybrid (i) & (ii). With regard to language-independent TC, our study relates to the statistical aspect only. The nature of textual data pre-processing includes: Document-base Representation (DR) and Feature Selection (FS). In this paper, we propose a hybrid statistical FS approach that integrates two existing (statistical FS) techniques, DIAAF (Darmstadt Indexing Approach Association Factor) and GSSC (Galavotti⋅Sebastiani⋅Simi Coefficient). Our proposed approach is presented under a statistical "bag of phrases" DR setting. The experimental results, based on the well-established associative text classification approach, demonstrate that our proposed technique outperforms existing mechanisms with respect to the accuracy of classification.

Decision Model for Optimization of Coagulation/Flocculation Process for Wastewater Treatment

“…The data pre-processing is an important step to improve the discernment precision of network, which can increase the forecasting capability and is similar to adjusting expression levels measured by northern analysis [15]. However, it is often neglected in the data mining process.…”