Scalable robust covariance and correlation estimates for data mining

Alqallaf, Fatemah; Konis, Kjell; Martin, R. Douglas; Zamar, Ruben H.

doi:10.1145/775047.775050

Cited by 51 publications

(25 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, robust correlation measures can be used to construct multivariate covariance matrices, based on pairwise covariances (see Kettering 1972, andMaronna andZamar 2002). For instance, Alqallaf et al (2002) use the Quadrant correlation to get a robust scatter matrix in very high dimensions. The resulting multivariate is highly robust and very fast to compute.…”

Section: Resultsmentioning

confidence: 99%

Influence functions of the Spearman and Kendall correlation measures

Croux

Dehon²

2010

Stat Methods Appl

480

202

View full text Add to dashboard Cite

Nonparametric correlation estimators as the Kendall and Spearman correlation are widely used in the applied sciences. They are often said to be robust, in the sense of being resistant to outlying observations. In this paper we formally study their robustness by means of their influence functions and gross-error sensitivities. Since robustness of an estimator often comes at the price of an increased variance, we also compute statistical efficiencies at the normal model. We conclude that both the Spearman and Kendall correlation estimators combine a bounded and smooth influence function with a high efficiency. In a simulation experiment we compare these nonparametric estimators with correlations based on a robust covariance matrix estimator.

show abstract

Section: Resultsmentioning

confidence: 99%

Influence functions of the Spearman and Kendall correlation measures

Croux

Dehon²

2010

Stat Methods Appl

480

202

View full text Add to dashboard Cite

show abstract

“…In recognition of this opportunity, [10] and [2] recently proposed new pairwise methods based on a modification of approaches (iii) and (ii) respectively, that preserve positive definiteness and have computational complexity O(np 2 ). However, these pairwise methods are not affine equivariant and may be upset by the so called two-dimensional structural outliers.…”

Section: Related Workmentioning

confidence: 99%

“…Our choice for c in the code was 0.00001. In actuality, by our ψ function, we are using a Huberized estimator, which in the limiting case is Quadrant Correlation [2]. The limiting case here would be to use the sign function in the place of ψ.…”

Section: Quadrant Correlationmentioning

confidence: 99%

See 1 more Smart Citation

Parallel Computation of High-Dimensional Robust Correlation and Covariance Matrices

Chilson

Wagner

et al. 2006

Algorithmica

View full text Add to dashboard Cite

The computation of covariance and correlation matrices are critical to many data mining applications and processes. Unfortunately the classical covariance and correlation matrices are very sensitive to outliers. Robust methods, such as QC and the Maronna method, have been proposed. However, existing algorithms for QC only give acceptable performance when the dimensionality of the matrix is in the hundreds; and the Maronna method is rarely used in practise because of its high computational cost.In this paper, we develop parallel algorithms for both QC and the Maronna method. We evaluate these parallel algorithms using a real data set of the gene expression of over 6,000 genes, giving rise to a matrix of over 18 million entries. In our experimental evaluation, we explore scalability in dimensionality and in the number of processors, and the trade-offs between accuracy and computational efficiency. We also compare the parallel behaviours of the two methods. From a statistical standpoint, the Maronna method is more robust than QC. From a computational standpoint, while QC requires less computation, interestingly the Maronna method is much more parallelizable than QC. After a thorough experimentation, we conclude that for many data mining applications, both QC and Maronna are viable options. Less robust, but faster, QC is the recommended choice for small parallel platforms. On the other hand, the Maronna method is the recommended choice when a high degree of robustness is required, or when the parallel platform features a large number of processors (e.g., 32).

show abstract

“…However, three-or higher-dimensional outliers may not be detected by univariate and bivariate analyses. Khan, Van Aelst, and Zamar 6 mentioned that the correlation matrix obtained from the pairwise correlation approach may not be positive definite, forcing the use of a correction for positive definiteness in some cases 7 . These problems have motivated us to improve this strategy by using a fast and robust multivariate location and dispersion that is robust to multivariate outliers.…”

Section: Introductionmentioning

confidence: 99%

Robust multivariate least angle regression

Uraibi¹,

Midi²,

Rana³

2017

ScienceAsia

View full text Add to dashboard Cite

ABSTRACT:The least angle regression selection (LARS) algorithms that use the classical sample means, variances, and correlations between the original variables are very sensitive to the presence of outliers and other contamination. To remedy this problem, a simple modification of this algorithm is to replace the non-robust estimates with their robust counterparts. Khan, Van Aelst, and Zamar employed the robust correlation for winsorized data based on adjusted winsorization correlation as a robust bivariate correlation approach for plug-in LARS. However, the robust least angle regression selection has some drawbacks in the presence of multivariate outliers. We propose to incorporate the Olive and Hawkins reweighted and fast consistent high breakdown estimator into the robust plug-in LARS method based on correlations. Our proposed method is tested by using a numerical example and a simulation study.

show abstract

Scalable robust covariance and correlation estimates for data mining

Cited by 51 publications

References 19 publications

Influence functions of the Spearman and Kendall correlation measures

Influence functions of the Spearman and Kendall correlation measures

Parallel Computation of High-Dimensional Robust Correlation and Covariance Matrices

Robust multivariate least angle regression

Contact Info

Product

Resources

About