Algorithm for the detection of outliers based on the theory of rough sets

Abstract

Outliers are objects that show abnormal behavior with respect to their context or that have unexpected values in some of their parameters. In decision-making processes, information quality is of the utmost importance. In specific applications, an outlying data element may represent an important deviation in a production process or a damaged sensor. Therefore, the ability to detect these elements could make the difference between making a correct or an incorrect decision. This task is complicated by the large sizes of typical databases. Due to their importance in search processes in large volumes of data, researchers pay special attention to the development of efficient outlier detection techniques. This article presents a computationally efficient algorithm for the detection of outliers in large volumes of information. This proposal is based on an extension of the mathematical framework upon which the basic theory of detection of outliers, founded on Rough Set Theory, has been constructed. From this starting point, current problems are analyzed; a detection method is proposed, along with a computational algorithm that allows the performance of outlier detection tasks with an almost-linear complexity. To illustrate its viability, the results of the application of the outlier-detection algorithm to the concrete example of a large database are presented. 1.-Introduction Decision support systems are computer-based programs that assist decision makers in effective and efficient decision-making. The proper functioning of these systems requires large amounts of precise, high-quality data. However, if the data contain abnormal, unrealistic or simply erroneous elements, it may misguide the decision-making process, thereby leading to incorrect Highlights • We propose a formal expansion to the theory of rough sets. • We propose an efficient algorithm for the detection of outliers. • We have implemented the algorithm and verified the theoretical results.

Detection of Outliers and Imputing of Missing Values for Water Quality UV-VIS Absorbance Time Series

“…Several value combinations of r and m parameters were verified, and the best results using r and m proved to be a value of ten (10). Although (30,50 or 70) for r and m parameters translate into less outlier values, the time series' shape becomes stair-like, losing its similarity to the shape of the original times series (Figure 3, graph a). Thus, using these large window size values would be disadvantageous because an analysis of the last part of the time series (roughly values 30950 until 31000) becomes largely impossible, as observed in Figure 3 (graphs d, e and f).…”

Application of the Variable Precision Rough Sets Model to Estimate the Outlier Probability of Each Element

“…However, its computational implementation is complicated by its exponential order. An extension of the theoretical framework of the previous proposition is presented in [6], in which an outlier detection algorithm is implemented based on Pawlak rough sets-the Pawlak rough sets algorithm-with a nonexponential order of temporal and spatial complexity. In [6], a method for the detection of outliers has been proposed with a simple and rigorous theoretical setup, starting from a definition of outliers that is simple, intuitive, and computationally viable for large datasets.…”

“…An extension of the theoretical framework of the previous proposition is presented in [6], in which an outlier detection algorithm is implemented based on Pawlak rough sets-the Pawlak rough sets algorithm-with a nonexponential order of temporal and spatial complexity. In [6], a method for the detection of outliers has been proposed with a simple and rigorous theoretical setup, starting from a definition of outliers that is simple, intuitive, and computationally viable for large datasets. From this method, an efficient algorithm for outlier mining has been developed, conceptually based on a novel and original approach using rough set theory, which has not been applied in any previous category of classification for the methods of rough set detection.…”

Recent Fuzzy Generalisations of Rough Sets Theory: A Systematic Review and Methodological Critique of the Literature

“…Though it is very likely that there are two e-mails that have the similar words among the list of 10 words, however for which one is spam and the other is nonspam. In this case, the equivalence relation is not A universe partitioned by an equivalence relation and a concept ⊆ that cannot be defined using [41,42].…”