Fast Top-k Distance-Based Outlier Detection on Uncertain Data

Shaikh, Salman Ahmed; Kitagawa, Hiroyuki

doi:10.1007/978-3-642-38562-9_31

Cited by 4 publications

(3 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Remark 1. Abnormal data detection based on the Gaussian distribution is a popular method [43]. The main idea can be described through a simple statement.…”

Section: Abnormal Data Identificationmentioning

confidence: 99%

A Bidirectional Searching Strategy to Improve Data Quality Based on K-Nearest Neighbor Approach

Liang

Qin

2019

Symmetry

View full text Add to dashboard Cite

Traffic data are the basis of traffic control, planning, management, and other implementations. Incomplete traffic data that are not conducive to all aspects of transport research and related activities can have adverse effects such as traffic status identification error and poor control performance. For intelligent transportation systems, the data recovery strategy has become increasingly important since the application of the traffic system relies on the traffic data quality. In this study, a bidirectional k-nearest neighbor searching strategy was constructed for effectively detecting and recovering abnormal data considering the symmetric time network and the correlation of the traffic data in time dimension. Moreover, the state vector of the proposed bidirectional searching strategy was designed based the bidirectional retrieval for enhancing the accuracy. In addition, the proposed bidirectional searching strategy shows significantly more accuracy compared to those of the previous methods.Symmetry 2019, 11, 815 2 of 18 and model parameter selection can be shown. In Section 4, basic KNN approach will be introduced. In Section 5, a novel bidirectional data recovery approach is proposed, and the parameter setting process is presented. Section 6 discusses the experimental design and the results. Section 7 concludes the paper with a summary of bidirectional searching strategy and gives the suggestions for future work. Literature ReviewSeveral works focusing on recovering the abnormal data response to the data quality control strategy. These studies examine the effect of abnormal data and research the data recovery approach based on the historical aiming to improve data quality. Pushkar et al. [17] applied the catastrophe theory to establish the three-parameter sudden change surface of traffic flow to recover data and then proposed the speed estimation method. A nearest-neighbor imputation algorithm was developed and applied to interpolate the missing data based on the average value of historical data at the same time interval [18]. In addition, the factor approach estimated the missing data by using the mean value of the key factors selected from the historical set [19]. Troyanskaya et al. [20] investigated automated methods for estimating missing data to minimize the effect of incomplete data sets on analyses. Smith et al. [21] performed a preliminary analysis of several heuristic and statistical imputation techniques and declared the statistical techniques are more accurate. Chen et al. [22] proposed a linear regression algorithm to impute missing or bad traffic flow data and occupancy data using neighboring sensors data. Abdella et al. [23] proposed an integrated method combining the genetic algorithms with neural networks aiming at seeking the approximating missing data in a database. Tang et al.[24] developed a hybrid approach integrating Fuzzy C-Means-based (FCM) imputation method with a genetic algorithm (GA) to estimate the missing traffic volume data based on inductance loop sensor outputs. This approach ...

show abstract

“…Remark 1. Abnormal data detection based on the Gaussian distribution is a popular method [43]. The main idea can be described through a simple statement.…”

Section: Abnormal Data Identificationmentioning

confidence: 99%

A Bidirectional Searching Strategy to Improve Data Quality Based on K-Nearest Neighbor Approach

Liang

Qin

2019

Symmetry

View full text Add to dashboard Cite

show abstract

“…Moreover, no outlier ranking is available and users are unable to differentiate between strong and weak outliers. Therefore, we presented a top-k approach of distance-based outliers in one of our recent works [34] . The proposed approach [34] returns k objects with lowest outlier scores (#D-neighbors), in other words, k strongest outliers along with their ranking.…”

Section: Related Workmentioning

confidence: 99%

“…This paper is an extended version of our recent work [34] . The main contributions of this paper include a bounded Gaussian approach of the top-k outlier detection presented in Section 6, complexity analysis of the proposed top-k algorithms (Section 5.3), discussion on the determination of values for parameters D and l (Section 5.4), detailed experiments comparing the accuracy of the proposed approach with the deterministic approach of outlier detection by Knorr et al [10] and an extensive empirical study on performance using larger real and synthetic datasets.…”

Section: Related Workmentioning

confidence: 99%

Top-k Outlier Detection from Uncertain Data

Shaikh

Kitagawa

2014

Int. J. Autom. Comput.

Self Cite

View full text Add to dashboard Cite

Uncertain data are common due to the increasing usage of sensors, radio frequency identification (RFID), GPS and similar devices for data collection. The causes of uncertainty include limitations of measurements, inclusion of noise, inconsistent supply voltage and delay or loss of data in transfer. In order to manage, query or mine such data, data uncertainty needs to be considered. Hence, this paper studies the problem of top-k distance-based outlier detection from uncertain data objects. In this work, an uncertain object is modelled by a probability density function of a Gaussian distribution. The naive approach of distance-based outlier detection makes use of nested loop. This approach is very costly due to the expensive distance function between two uncertain objects. Therefore, a populated-cells list (PC-list) approach of outlier detection is proposed. Using the PC-list, the proposed top-k outlier detection algorithm needs to consider only a fraction of dataset objects and hence quickly identifies candidate objects for top-k outliers. Two approximate top-k outlier detection algorithms are presented to further increase the efficiency of the top-k outlier detection algorithm. An extensive empirical study on synthetic and real datasets is also presented to prove the accuracy, efficiency and scalability of the proposed algorithms.

show abstract