Transforming big data into smart data: An insight on the use of the k‐nearest neighbors algorithm to obtain quality data

Abstract

The k‐nearest neighbors algorithm is characterized as a simple yet effective data mining technique. The main drawback of this technique appears when massive amounts of data—likely to contain noise and imperfections—are involved, turning this algorithm into an imprecise and especially inefficient technique. These disadvantages have been subject of research for many years, and among others approaches, data preprocessing techniques such as instance reduction or missing values imputation have targeted these weaknesses. As a result, these issues have turned out as strengths and the k‐nearest neighbors rule has become a core algorithm to identify and correct imperfect data, removing noisy and redundant samples, or imputing missing values, transforming Big Data into Smart Data—which is data of sufficient quality to expect a good outcome from any data mining algorithm. The role of this smart data gleaning algorithm in a supervised learning context are investigated. This includes a brief overview of Smart Data, current and future trends for the k‐nearest neighbor algorithm in the Big Data context, and the existing data preprocessing techniques based on this algorithm. We present the emerging big data‐ready versions of these algorithms and develop some new methods to cope with Big Data. We carry out a thorough experimental analysis in a series of big datasets that provide guidelines as to how to use the k‐nearest neighbor algorithm to obtain Smart/Quality Data for a high‐quality data mining process. Moreover, multiple Spark Packages have been developed including all the Smart Data algorithms analyzed. This article is categorized under: Technologies > Data Preprocessing Fundamental Concepts of Data and Knowledge > Big Data Mining Technologies > Classification

Monitoring Asynchrony During Invasive Mechanical Ventilation

“…127 To enable individualized assessment using physiological biomarkers to predict whether a particular patient is likely to experience an adverse event, systems must be capable of automatically transforming raw data into useful information and presenting it in integrated displays showing the information that is most relevant for clinical decision making. 124,[127][128][129] Alert thresholds derived from continuous predictive analytical monitoring can be operationalized as a degree of change from the patient's own baseline rather than from arbitrary cutoffs. 130 To develop smart alarms, artificial intelligence can incorporate multiple datasets, including demographics, mechanical ventilation data, and medical texts (structured or unstructured).…”

Virtual learning environment to predict withdrawal by leveraging deep learning

“…This data attained from various activities can be investigated to evaluate the potential behavior of learners, examining interactional activities of students and tracing different interaction patterns of successful and at‐risk students, proposing similar counteractive measures based on learner's performances, consequently supporting instructors in improvising the pedagogical practices . These repositories encompass raw data that needs to be transformed into a meaningful interpretation, and a more quality enhanced form, to extract valuable insights and patterns …”

From Big to Smart Data: Iterative ensemble filter for noise filtering in Big Data classification

“…Smart Data aims to separate the raw (or Big) part of the data (volume/velocity), from the Smart part of it (veracity/value) . Therefore, Smart Data is focused on extracting valuable knowledge from data, in the form of a subset, that contains enough quality for a successful data mining process …”

“…That is why there is a special need for noise filters in Big Data. Although we can find many proposals for dealing with noise for normal‐sized data in the literature, in Big Data scenarios we can find only a handful of proposals devoted to this problem …”