Redundancy and Complexity Metrics for Big Data Classification: Towards Smart Data

Abstract

It is recognized the importance of knowing the descriptive properties of a dataset when tackling a data science problem. Having information about the redundancy, complexity and density of a problem allows us to make decisions as to which data preprocessing and machine learning techniques are most suitable. In classification problems, there are multiple metrics to describe the overlapping of the features between classes, class imbalances or separability, among others. However, these metrics may not scale up well when dealing with big datasets, or may not simply be sufficiently informative in this context. In this paper, we provide a package of metrics for big data classification problems. In particular, we propose two new big data metrics: Neighborhood Density and Decision Tree Progression, which study density and accuracy progression by discarding half of the samples. In addition, we enable a number of basic metrics to handle big data. The experimental study carried out in standard big data classification problems shows that our metrics can quickly characterize big datasets. We identified a clear redundancy of information in most datasets, so that, discarding randomly 75% of the samples does not drastically affect the accuracy of the classifiers used. Thus, the proposed big data metrics, which are available as a Spark-Package, provide a fast assessment of the shape of a classification dataset prior to applying big data preprocessing, toward smart data.

FDR2-BD: A Fast Data Reduction Recommendation Tool for Tabular Big Data Classification Problems

“…Regardless of which data reduction method is applied, the actual benefits of its application in the data science life-cycle are alleviating storage and memory requirements, the complexity of algorithmic computation, and runtimes, in addition to the fact that they may obtain better quality data by removing conceptual redundancy [9].…”

“…For this purpose, different paradigms of classification algorithms exist, and they are used in multiple application fields [7,8]. When analyzing the publicly available tabular big data problems for binary classification, the presence of a notable conceptual redundancy of information in the data might be observed [9] (redundant instances and/or features) that leads to an unnecessary computational cost. Furthermore, it is known that the classifiers only need a set of instances that correctly represent a problem in order to generate an adequate model [10].…”

“…Recent studies have shown that publicly available big datasets used as benchmarks might have questionable quality regarding some unwanted aspects [9,21], such as redundant examples, disproportion of classes, and noise, among others. The presence of some of these issues could be due to the nature of the data or to the artificial generation of the data by replication of instances in order to generate big datasets.…”

Quantitative Analysis of a Weak Correlation between Complicated Data on the Basis of Principal Component Analysis

“…The reason is that there are many different kinds of data types or sources, and the unknown relationship between various variables is quite complicated and not easy to be discovered. This leads to the difficulty for reasonable interpretation on the basis of the conventional methods for univariate or multivariate analysis [ 9 , 10 ].…”

Nuevos Algoritmos Basados en Grafos y Clustering para el Tratamiento de Complejidades de los Datos

“…La falta de métodos de bajo-muestreo para enfrentar el problema de desbalance de clases en contexto de Big Data deja una amplia brecha para desarrollar propuestas. Al respecto, la reducción del tamaño del conjunto de datos mediante la selección inteligente de instancias permite tener un mejor rendimiento en modelos de aprendizaje con la característica de usar un número reducido de datos, que de acuerdo al estudio presentado por Maillo et al [89] no es indispensable contar con un número elevado de instancias para generar resultados de clasificación elevados.…”