A Nonparametric Outlier Detection for Effectively Discovering Top-N Outliers from Engineering Data

Fan, Hongqin; Zaı̈ane, Osmar R.; Foss, Andrew; Wu, Jian

doi:10.1007/11731139_66

Cited by 53 publications

(37 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Furthermore, an exact algorithm is introduced for outlier detection based on the LOCI model. The resolution-based outlier factor (ROF) [9] is a mix of the local and the global outlier paradigm. The outlier schema is based on the idea of a change of resolution.…”

Section: Related Workmentioning

confidence: 99%

Angle-based outlier detection in high-dimensional data

Kriegel

Schubert

Zimek

2008

Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

649

403

View full text Add to dashboard Cite

Detecting outliers in a large set of data objects is a major data mining task aiming at finding different mechanisms responsible for different groups of objects in a data set. All existing approaches, however, are based on an assessment of distances (sometimes indirectly by assuming certain distributions) in the full-dimensional Euclidean data space. In high-dimensional data, these approaches are bound to deteriorate due to the notorious "curse of dimensionality". In this paper, we propose a novel approach named ABOD (Angle-Based Outlier Detection) and some variants assessing the variance in the angles between the difference vectors of a point to the other points. This way, the effects of the "curse of dimensionality" are alleviated compared to purely distance-based approaches. A main advantage of our new approach is that our method does not rely on any parameter selection influencing the quality of the achieved ranking. In a thorough experimental evaluation, we compare ABOD to the well-established distance-based method LOF for various artificial and a real world data set and show ABOD to perform especially well on high-dimensional data.

show abstract

Section: Related Workmentioning

confidence: 99%

Angle-based outlier detection in high-dimensional data

Kriegel

Schubert

Zimek

2008

Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

649

403

View full text Add to dashboard Cite

show abstract

“…Therefore, to improve overall performances and accuracy, it has become necessary to develop data mining algorithms using the whole data distribution as well as most of data features [10]. In this paper, we focus on clustering-based outlier detection algorithms [10,7,6,16]. Such techniques rely on the assumption that normal points belong to large clusters while outliers either do not belong to any cluster [10] or form very small and tight clusters [7].…”

Section: Related Workmentioning

confidence: 99%

“…In this paper, we focus on clustering-based outlier detection algorithms [10,7,6,16]. Such techniques rely on the assumption that normal points belong to large clusters while outliers either do not belong to any cluster [10] or form very small and tight clusters [7]. In other words, outlier detection consists in identifying among data those that are far from being significant clusters.…”

Section: Related Workmentioning

confidence: 99%

Data Mining for Intrusion Detection: From Outliers to True Intrusions

Singh

Masséglia

Fiot

et al. 2009

Advances in Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

Abstract. Data mining for intrusion detection can be divided into several sub-topics, among which unsupervised clustering has controversial properties. Unsupervised clustering for intrusion detection aims to i) group behaviors together depending on their similarity and ii) detect groups containing only one (or very few) behaviour. Such isolated behaviours are then considered as deviating from a model of normality and are therefore considered as malicious. Obviously, all atypical behaviours are not attacks or intrusion attempts. Hence, this is the limits of unsupervised clustering for intrusion detection. In this paper, we consider to add a new feature to such isolated behaviours before they can be considered as malicious. This feature is based on their possible repetition from one information system to another. We propose a new outlier mining principle and validate it through a set of experiments.

show abstract

“…Density-based techniques [8,16,59,46,87,63,33,62] are proposed to take the local density into account when searching for outliers. The computation of density still depends on full dimensional distance measure between a point and its nearest neighbors in a dataset.…”

Section: Shortcomings Of General Outlier Detection Techniquesmentioning

confidence: 99%

Observing the unobservable : distributed online outlier detection in wireless sensor networks

Zhang¹

View full text Add to dashboard Cite

This research was conducted within the EU projects e-SENSE and SENSEI. Copyright c 2010 by Yang Zhang, Enschede, The Netherlands. All rights reserved. No part of this book may be reproduced or transmitted, in any form or by any means, electronic or mechanical, including photocopying, microfilming, and recording, or by any information storage or retrieval system, without the prior written permission of the author.Printed by Wöhrmann Print Service. AbstractThe generation of wireless sensor networks (WSNs) makes human beings observe and reason about the physical environment better, easier, and faster. The wireless sensor nodes equipped with sensing, processing, wireless communication and actuation capabilities can be densely deployed in a wide geographical area and measure various parameters continuously from the physical world. Compared with traditional environmental sensing technologies, such densely deployed WSNs enable collection of fine-grained high spatial and temporal resolution data with less installation, maintenance, and operation costs.However, raw sensor observations often have low data quality and reliability due to both internal and external factors including low quality of cheap sensors, dynamicity of network conditions, and harshness of the deployment environment. Use of low quality sensor data in any data analysis and decision making process will not only negatively impact analysis results and decisions made but also waste huge amount of valuable and limited network resources such as energy, as many incorrect values are transmitted. Low quality sensor data also prevents WSNs to fulfill their promises in terms of reliable real-time situation-awareness, as the low quality sensor data may generate large number of false alarms.Motivated by the need to improve quality of data analysis and decision making, enhance efficiency of using WSNs resources by preventing unnecessary transmission of erroneous sensor observations, and increase effectiveness of monitoring and situation-awareness capabilities of the WSNs, in this thesis we focus on online identification of outliers whenever and wherever they occur. Outliers in WSNs are those observations that represent erroneous values (errors) or indicate particular phenomenal changes (events). Our outlier detection techniques, which are based on distributed in-network data processing, identify sensor observations that do not conform to normal behavior of sensor data without using a pre-defined threshold or triggering conditions.Our main research objective is to design and implement effective and efficient outlier detection techniques for WSNs to identify outliers in an online and disv tributed manner and distinguish between errors and events with high accuracy and low false alarm, while maintaining the communication, computation and memory complexity low. Main contributions of this thesis can be summarized as: 3. Statistical-Based outlier detection techniques for WSNs. We take two approaches in designing our outlier detection techniques. One approach originates fr...

show abstract

A Nonparametric Outlier Detection for Effectively Discovering Top-N Outliers from Engineering Data

Cited by 53 publications

References 9 publications

Angle-based outlier detection in high-dimensional data

Angle-based outlier detection in high-dimensional data

Data Mining for Intrusion Detection: From Outliers to True Intrusions

Observing the unobservable : distributed online outlier detection in wireless sensor networks

Contact Info

Product

Resources

About