Parallel computation of high dimensional robust correlation and covariance matrices

Abstract

The computation of covariance and correlation matrices are critical to many data mining applications and processes. Unfortunately the classical covariance and correlation matrices are very sensitive to outliers. Robust methods, such as QC and the Maronna method, have been proposed. However, existing algorithms for QC only give acceptable performance when the dimensionality of the matrix is in the hundreds; and the Maronna method is rarely used in practise because of its high computational cost. In this paper, we develop parallel algorithms for both QC and the Maronna method. We evaluate these parallel algorithms using a real data set of the gene expression of over 6,000 genes, giving rise to a matrix of over 18 million entries. In our experimental evaluation, we explore scalability in dimensionality and in the number of processors, and the trade-offs between accuracy and computational efficiency. We also compare the parallel behaviours of the two methods. From a statistical standpoint, the Maronna method is more robust than QC. From a computational standpoint, while QC requires less computation, interestingly the Maronna method is much more parallelizable than QC. After a thorough experimentation, we conclude that for many data mining applications , both QC and Maronna are viable options. Less robust, but faster, QC is the recommended choice for small parallel platforms. On the other hand, the Maronna method is the recommended choice when a high degree of robustness is required, or when the parallel platform features a large number of processors (e.g., 32).

Comparison study on the speed of convergence in distribution of sample generalized variance and sample vector variance

“…In this case, some value of GV cannot be compute since covariance matrix is not positive definite anymore and it becomes singular matrix. See [3], [19], [20] and [21] for further discussion regarding this problem.…”

Machine Learning-Based Anomaly Detection on Seawater Temperature Data with Oversampling

“…Research on unsupervised anomaly detection techniques has been actively conducted using various approaches. In the early days of the study, distribution-based anomaly detection techniques [40,52], depth-based anomaly detection techniques [53], and clustering-based anomaly detection techniques [51] have been mainly studied, but recent research trends largely utilize distanceand density-based anomaly detection techniques [54].…”

Some Methods for Detecting Outliers

“…A data cloud is described by a finite number of depth contours in halfspace depth. [23] Rousseeuw and Struyf (1998) introduce an algorithm to compute the halfspace depth for d = 3 with complexity O(n 2 log n). They use the Rousseeuw and Ruts (1999) procedure to determine the halfspace depth in these planes by projecting points onto planes orthogonal to the lines linking each of the points from X with z.…”