Evaluation of robust outlier detection methods for zero-inflated complex data

Abstract

Outlier detection can be seen as a pre-processing step for locating data points in a data sample, which do not conform to the majority of observations. Various techniques and methods for outlier detection can be found in the literature dealing with different types of data. However, many data sets are inflated by true zeros and, in addition, some components/variables might be of compositional nature. Important examples of such data sets are the Structural Earnings Survey, the Structural Business Statistics, the European Statistics on Income and Living Conditions, tax data or-as in this contributionhousehold expenditure data which are used, for example, to estimate the Purchase Power Parity of a country. In this work, robust univariate and multivariate outlier detection methods are compared by a complex simulation study that considers various challenges included in data sets, namely structural (true) zeros, missing values, and compositional variables. These circumstances make it difficult or impossible to flag true outliers and influential observations by well-known outlier detection methods. Our aim is to assess the performance of outlier detection methods in terms of their effectiveness to identify outliers when applied to challenging data sets such as the household expenditures data surveyed all over the world. Moreover, different methods are evaluated through a close-to-reality simulation study. Differences in performance of univariate and multivariate robust techniques for outlier detection and their shortcomings are reported. We found that robust multivariate methods outperform robust univariate methods. The best performing methods in finding the outliers and in providing a low false discovery rate were found to be the generalized S estimators (GSE), the BACON-EEM algorithm and a compositional method (CoDa-Cov). In addition, these methods performed also best when the outliers are imputed based on the corresponding outlier detection method and indicators are estimated from the data sets.

Cognitive impairment in schizophrenia: relationships with cortical thickness in fronto-temporal regions, and dissociability from symptom severity

“…Cognitive scores were screened and individuals with outliers in any domain were excluded based on Interquartile Range (IQR): criteria for exclusion were values below Q1-1.5 IQR or above Q3 + 1.5 IQR, box-and-whisker plots were used 78 , 79 (also known as Tukey’s method 80 , 81 ). After removing individuals with outliers the final sample consisted of 70 SZH (18 females, aged between 19 and 65, mean age 37.56, see Table 1 ) and 72 age and gender-matched healthy controls (18 females, aged between 18 and 65, mean age 38.44, see Table 1 ).…”

Estimated prevalence of eating disorders in Singapore

“…The distributions for BMI and ED behaviors were positively skewed. As height and weight values were self‐reported, we used the Median Absolute Deviation (MAD) method (Leys, Ley, Klein, Bernard, & Licata, 2013), with a conservative cutoff of ±3 times the MAD to detect outliers (Templ, Gussenbauer, & Filzmoser, 2019). Based on the MAD cutoffs, 2.7 and 1.8% of weight and height values were considered implausible and considered missing.…”

Conditional Income Disparity between Farm and Non‐farm Households in the European Union: A Longitudinal Analysis

“…Moreover, to overcome the problem of outliers and influential observations, we recalibrate sample weights following the approach proposed by Alfons et al . (2013) and generally adopted by those working with income variables (Alfons and Templ, 2013; Brzesinki, 2016; Jenkins, 2017; Safari et al ., 2018, 2019; Templ et al ., 2019). This procedure consists of detecting outlier observations against a fitted Pareto distribution of the variable of interest, applying Van Kerm’s rule of thumb to determine the threshold (Van Kerm, 2007).…”