Scalable robust covariance and correlation estimates for data mining

Alqallaf, Fatemah; Konis, Kjell; Martin, R. Douglas; Zamar, Ruben H.

doi:10.1145/775048.775050

Cited by 24 publications

(27 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this paradigm a few cells in a row (case) can be anomalous whereas many other cells in the same row still contain useful information, and in such situations we would rather not remove or downweight the entire row. The cellwise framework was first proposed and studied by Alqallaf et al (2002Alqallaf et al ( , 2009).…”

Section: Fast Detection Of Anomalous Cellsmentioning

confidence: 99%

Fast Robust Correlation for High-Dimensional Data

Raymaekers

Rousseeuw

2019

Technometrics

View full text Add to dashboard Cite

The product moment covariance matrix is a cornerstone of multivariate data analysis, from which one can derive correlations, principal components, Mahalanobis distances and many other results. Unfortunately the product moment covariance and the corresponding Pearson correlation are very susceptible to outliers (anomalies) in the data. Several robust estimators of covariance matrices have been developed, but few are suitable for the ultrahigh dimensional data that are becoming more prevalent nowadays. For that one needs methods whose computation scales well with the dimension, are guaranteed to yield a positive semidefinite matrix, and are sufficiently robust to outliers as well as sufficiently accurate in the statistical sense of low variability. We construct such methods using data transformations. The resulting approach is simple, fast and widely applicable. We study its robustness by deriving influence functions and breakdown values, and computing the mean squared error on contaminated data. Using these results we select a method that performs well overall. This also allows us to construct a faster version of the DetectDeviatingCells method (Rousseeuw and Van den Bossche, 2018) to detect cellwise outliers, that can deal with much higher dimensions. The approach is illustrated on genomic data with 12,600 variables and color video data with 920,000 dimensions.

show abstract

Section: Fast Detection Of Anomalous Cellsmentioning

confidence: 99%

Fast Robust Correlation for High-Dimensional Data

Raymaekers

Rousseeuw

2019

Technometrics

View full text Add to dashboard Cite

show abstract

“…Alternatively, pairwise scatter estimators could be used as fast initial estimator (e.g., Alqallaf et al, 2002). Previous simulation studies have shown that pairwise scatter estimators are robust against cellwise outliers, but they perform not as well in the presence of casewise outliers and finely shaped multivariate data (Danilov et al, 2012;Agostinelli et al, 2015b).…”

Section: Computation Of the Initial Estimatormentioning

confidence: 99%

Multivariate location and scatter matrix estimation under cellwise and casewise contamination

Leung

Yohai

Zamar

2017

Computational Statistics & Data Analysis

View full text Add to dashboard Cite

We consider the problem of multivariate location and scatter matrix estimation when the data contain cellwise and casewise outliers. Agostinelli et al. (2015b) propose a two-step approach to deal with this problem: first apply a univariate filter to remove cellwise outliers and second apply a generalized S-estimator to downweight casewise outliers. We improve this proposal in three main directions. First, we introduce a consistent bivariate filter to be used in combination with the univariate filter in the first step. Second, we propose a new fast subsampling procedure to generate starting points for the generalized S-estimator in the second step. Third, we consider a non-monotonic weight function for the generalized S-estimator to better deal with casewise outliers in high dimension. A simulation study and real data example show that, unlike the original two-step procedure, the modified two-step approach performs and scales well for high dimension. Moreover, the modified procedure outperforms the original one and other state of the art robust procedures under cellwise and casewise data contamination. arXiv:1609.00402v2 [math.ST] Abstract This supplementary material contains all the proofs, additional simulation results, and related supplementary material referenced in the article "Robust Estimation of Multivariate Location and Scatter in the Presence of Cellwise and Casewise Contamination".

show abstract

“…Following the notation of Alqallaf et al (2002Alqallaf et al ( , 2009, we write the cellwise contamination model in the following form:…”

Section: Background and Problem Setupmentioning

confidence: 99%

“…In addition, the MVE, MCD, and SD estimators all require heavy computational effort, rendering them impractical for high-dimensional datasets. To deal with cellwise contamination, Van Aelst (2014) proposed a modified SD estimator that adapts winsorization (Huber, 1981;Alqallaf et al, 2002) and a cellwise weighting scheme. Similar to the original SD estimator, however, computation is only feasible for small p. A recent approach by Agostinelli et al (2014) is capable of dealing with both rowwise and cellwise outliers.…”

Section: Introductionmentioning

confidence: 99%

High-dimensional robust precision matrix estimation: Cellwise corruption under $\epsilon $-contamination

Loh¹,

Tan²

2018

Electron. J. Statist.

View full text Add to dashboard Cite

We analyze the statistical consistency of robust estimators for precision matrices in high dimensions. We focus on a contamination mechanism acting cellwise on the data matrix. The estimators we analyze are formed by plugging appropriately chosen robust covariance matrix estimators into the graphical Lasso and CLIME. Such estimators were recently proposed in the robust statistics literature, but only analyzed mathematically from the point of view of the breakdown point. This paper provides complementary high-dimensional error bounds for the precision matrix estimators that reveal the interplay between the dimensionality of the problem and the degree of contamination permitted in the observed distribution. We also show that although the graphical Lasso and CLIME estimators perform equally well from the point of view of statistical consistency, the breakdown property of the graphical Lasso is superior to that of CLIME. We discuss implications of our work for problems involving graphical model estimation when the uncontaminated data follow a multivariate normal distribution, and the goal is to estimate the support of the population-level precision matrix. Our error bounds do not make any assumptions about the the contaminating distribution and allow for a nonvanishing fraction of cellwise contamination.

show abstract

Scalable robust covariance and correlation estimates for data mining

Cited by 24 publications

References 0 publications

Fast Robust Correlation for High-Dimensional Data

Fast Robust Correlation for High-Dimensional Data

Multivariate location and scatter matrix estimation under cellwise and casewise contamination

High-dimensional robust precision matrix estimation: Cellwise corruption under $\epsilon $-contamination

Contact Info

Product

Resources

About