Wild adaptive trimming for robust estimation and cluster analysis

Abstract

Trimming principles play an important role in robust statistics. However, their use for clustering typically requires some preliminary information about the contamination rate and the number of groups. We suggest a fresh approach to trimming that does not rely on this knowledge and that proves to be particularly suited for solving problems in robust cluster analysis. Our approach replaces the original K‐population (robust) estimation problem with K distinct one‐population steps, which take advantage of the good breakdown properties of trimmed estimators when the trimming level exceeds the usual bound of 0.5. In this setting, we prove that exact affine equivariance is lost on one hand but, on the other hand, an arbitrarily high breakdown point can be achieved by “anchoring” the robust estimator. We also support the use of adaptive trimming schemes, in order to infer the contamination rate from the data. A further bonus of our methodology is its ability to provide a reliable choice of the usually unknown number of groups.

On Characterizations and Tests of Benford’s Law

“…Following Bolton and Hand (2002), most unsupervised fraud detection methods look for anomalies in the data. Therefore, all of these techniques assume (either explicitly or implicitly) that the available data have been generated by a k-variate random vector, say Y, whose distribution function F Y is an (unknown) element within the following family C of distribution functions (see, e.g., Cerioli, Farcomeni, and Riani 2019, and the references therein)…”

“…A more accurate comparison would entail the use of formal outlier detection procedures for such complex structures, which are not yet available. One possible solution in this direction could be to build inferential statements for the robust clustering techniques adopted by Cerioli and Perrotta (2014) and by Cerioli, Farcomeni, and Riani (2019), which might be broadly regarded as cluster-wise extensions of LTS and FS, respectively. The development of suitable diagnostics and formal outlier labeling rules for clustered trade-data structures will be the subject of future research.…”

Semiautomatic robust regression clustering of international trade data

“…After discovering the number of groups, it is of interest to verify the quality of the classification. We have developed an approach that alternates (hopefully k times) the identification of an homogeneous sub-group using the random start approach and its subsequent elimination, following an idea initially explored in Torti (2011) and Cerioli et al (2019). This approach replaces the original k population (robust) estimation problem with k distinct one-population steps, which take advantage of the good breakdown properties of trimmed estimators when the trimming level exceeds the usual bound of 0.5.…”

Robust inference for parsimonious model-based clustering

“…Several other possibilities are listed in Farcomeni and Greco [3], but alternatives are difficult to implement or similarly lead to loss of affine invariance. See also Coretto and Hennig [7] and Cerioli et al [8].…”