Improving the performance of Naive Bayes multinomial in e-mail foldering by introducing distribution-based balance of datasets

Bermejo, Pablo; Gámez, José A.; Puerta, José M.

doi:10.1016/j.eswa.2010.07.146

Cited by 64 publications

(33 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The This problem present in many real world problems such as medical diagnosis [8], fraud detection, finances, risk management, network intrusion, E-mail foldering [12], Software Defect Detection [18] and so on. Additionally, the positive (minority) class is the class of interest from the learning point of view and has great impact when it is not classified properly.…”

Section: A Imbalanced Data Problemmentioning

confidence: 99%

Enhanced SMOTE algorithm for classification of imbalanced big-data using Random Forest

Bhagat¹,

Patil²

2015

2015 IEEE International Advance Computing Conference (IACC)

View full text Add to dashboard Cite

In the era of big data, the applications generating tremendous amount of data are becoming the main focus of attention as the wide increment of data generation and storage that has taken place in the last few years. This scenario is challenging for data mining techniques which are not arrogated to the new space and time requirements. In many of the real world applications, classification of imbalanced data-sets is the point of attraction. Most of the classification methods focused on two-class imbalanced problem. So, it is necessary to solve multi-class imbalanced problem, which exist in real-world domains.In the proposed work, we introduced a methodology for classification of multi-class imbalanced data. This methodology consists of two steps: In first step we used Binarization techniques (OVA and OVO) for decomposing original dataset into subsets of binary classes. In second step, the SMOTE algorithm is applied against each subset of imbalanced binary class in order to get balanced data. Finally, to achieve classification goal Random Forest (RF) classifier is used. Specifically, oversampling technique is adapted to big data using MapReduce so that this technique is able to handle as large data-set as needed.An experimental study is carried out to evaluate the performance of proposed method. For experimental analysis, we have used different datasets from UCI repository and the proposed system is implemented on Apache Hadoop and Apache Spark platform. The results obtained shows that proposed method outperforms over other methods.

show abstract

Section: A Imbalanced Data Problemmentioning

confidence: 99%

Enhanced SMOTE algorithm for classification of imbalanced big-data using Random Forest

Bhagat¹,

Patil²

2015

2015 IEEE International Advance Computing Conference (IACC)

View full text Add to dashboard Cite

show abstract

“…Un documento se trata como una secuencia de palabras y se supone que cada posición de la palabra se genera independientemente de cualquier otra. Es un clasificador rápido, fácil de implementar y relativamente eficaz [11].El modelo Naïve Bayes Multinomial permite considerar la frecuencia de aparición de cada término en los documentos, esto es importante, ya que podemos suponer que una alta frecuencia de aparición aumenta la probabilidad de pertenecer a una clase particular [1].…”

Section: Naïve Bayes Multinomialunclassified

“…Las máquinas de soporte vectorial en Python cuentan con métodos de aprendizaje supervisado utilizados para la clasificación, regresión y detección de valores atípicos. La herramienta scikit-learn[3] de Python es una máquina de aprendizaje en Python, que proporciona simples y eficientes herramientas para la minería de datos y análisis de sentimientos, contiene varios vectorizers 1 de traducción para los documentos de entrada en vectores de características 1. Función utilizada en scikit-learn http://scikit-learn.org/…”

unclassified

Análisis comparativo entre diferentes entornos de aprendizaje autmático para el análisis de sentimientos

Vazquez¹,

Tovar²,

Reyes‐Ortiz³

et al. 2016

RCS

View full text Add to dashboard Cite

Resumen. El objetivo en este artículo es el estudio del Análisis de Sentimientos con aprendizaje automático y precisamente realizar una comparativa de clasificadores automáticos implementados en la plataforma de software para el aprendizaje automático, WEKA y el lenguaje de programación Python. Los experimentos se elaboraron con tres conjuntos de datos provenientes de SemEval 2016 para la resolución de la tarea 5, subtarea 2 [9]. El enfoque utilizado para el análisis de sentimiento está dividido en 5 fases y los resultados muestran que Python obtiene un mejor desempeño con los clasificadores utilizados.Palabras clave: Análisis de sentimientos, procesamiento de lenguaje natural, aprendizaje autómatico.Abstract. The main aim of this paper is the study of Sentiment Analysis with machine learning techniques and to perform a comparison of automatic classifiers implemented in WEKA and Python programming language. The experiments were carried out with three data sets from SemEval 2016 for solving the task 5, subtask 2 [9]. The approach proposed for sentiment analysis is divided into five phases and the results show that Python achieves a better performance than WEKA with the classifiers used.

show abstract

“…Another common approach is to consider additional properties of email messages and utilize structural features, e.g., [2,15]. Results for these methods are commonly given for the Enron and SRI datasets [4,6,7,15].…”

Section: Related Workmentioning

confidence: 99%

“…The approaches include rule-based systems (e.g., [9,18]), IR methods: k-NN [6] and tf-idf, and machine learning methods: naive Bayes (NB), decision trees, support vector machines(SVM), maximum entropy (MaxEnt) and neural networks (NN) [1,4,10,14]. Another common approach is to consider additional properties of email messages and utilize structural features, e.g., [2,15].…”

Section: Related Workmentioning

confidence: 99%

Automatically tagging email by leveraging other users' folders

Koren¹,

Liberty²,

Maarek³

et al. 2011

Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

Most email applications devote a significant part of their real estate to organization mechanisms such as folders. Yet, we verified on the Yahoo! Mail service that 70% of email users have never defined a single folder. This implies that one of the most well known email features is underexploited. We propose here to revive the feature by providing a method for generating a lighter form of folders, or tags, benefiting even the most passive users. The method automatically associates, whenever possible, an appropriate semantic tag with a given email. This gives rise to an alternate mechanism for organizing and searching email.We advocate a novel modeling approach that exploits the overall population of users, thereby learning from the wisdomof-crowds how to categorize messages. Given our massive user base, it is enough to learn from a minority of the users who label certain messages in order to label that kind of messages for the general population. We design a novel cascade classification approach, which copes with the severe scalability and accuracy constraints we are facing. Significant efficiency gains are achieved by working within a low dimensional latent space, and by using a novel hierarchical classifier. Precision level is controlled by separating the task into a two-phase classification process.We performed an extensive empirical study covering three different time periods, over 100 million messages, and thousands of candidate tags per message. The results are encouraging and compare favorably with alternative approaches. Our method successfully tags 72% of incoming email traffic. Performance-wise, the computational overhead, even on surge large traffic, is sufficiently low for our approach to be applicable in production on any large Web mail service.

show abstract

Improving the performance of Naive Bayes multinomial in e-mail foldering by introducing distribution-based balance of datasets

Cited by 64 publications

References 31 publications

Enhanced SMOTE algorithm for classification of imbalanced big-data using Random Forest

Enhanced SMOTE algorithm for classification of imbalanced big-data using Random Forest

Análisis comparativo entre diferentes entornos de aprendizaje autmático para el análisis de sentimientos

Automatically tagging email by leveraging other users' folders

Contact Info

Product

Resources

About