FSDA: A MATLAB toolbox for robust analysis and interactive data exploration

Riani, Marco; Perrotta, Domenico; Torti, Francesca

doi:10.1016/j.chemolab.2012.03.017

Cited by 73 publications

(41 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1, which has to do with imports of "sports footwear with insoles of a length of less than 24 cm". We see the results obtained with the SAS PROC ROBUSTREG and the MATLAB FSDA Toolbox (Riani et al 2012(Riani et al , 2015 with four methods: FS, LTS, S and MM. We could verify that the reason for the discordance between the two sets of results originates from the different default parameter values adopted by the two environments.…”

Section: Monitoring Trade Datamentioning

confidence: 99%

Discussion of “The power of monitoring: how to make the most of a contaminated multivariate sample”

Perrotta

Torti

2018

Stat Methods Appl

Self Cite

View full text Add to dashboard Cite

We contribute to the discussion of an article where Andrea Cerioli, Marco Riani, Anthony Atkinson and Aldo Corbellini review the advantages of analyzing multivariate data by monitoring how the estimated model parameters change as the estimation parameters vary. The focus is on robust methods and their sensitivity to the nominal efficiency and breakdown point. In congratulating with the authors for the clear and stimulating exposition, we contribute to its discussion with an overview of what we experienced in applying the monitoring in our application domain.Keywords Forward Search · MM-estimation · S-estimation · Density estimation · Thinning · International trade data PrologueAndrea Cerioli, Marco Riani, Anthony Atkinson and Aldo Corbellini (hereafter CRAC) are passionate supporters of the data analysis approach proposed for this discussion (Cerioli et al. 2018), which consists in monitoring the model parameters estimated for a reasonable range of values of the key parameters of the estimation method, and selecting those producing the best results. Robust estimation algorithms depend on several tuning constants, producing effects that should be monitored. CRAC focus on the key parameters used to specify the maximum possible breakdown or efficiency to be achieved. We would like to complement their exposition with recent B Francesca Torti

show abstract

Section: Monitoring Trade Datamentioning

confidence: 99%

Discussion of “The power of monitoring: how to make the most of a contaminated multivariate sample”

Perrotta

Torti

2018

Stat Methods Appl

Self Cite

View full text Add to dashboard Cite

show abstract

“…We applied the S-estimator with the custom function ρ a for a = 0.2 to the geyser dataset, using the FSDA toolbox of Riani et al (2012). This does yield a fit to the main cluster, as illustrated by the tolerance ellipse in Figure 4.…”

Section: The Choice Of the Rho Functionmentioning

confidence: 99%

Discussion of “The power of monitoring: how to make the most of a contaminated multivariate sample”

2018

View full text Add to dashboard Cite

Abstract. This is an invited comment on the discussion paper "The power of monitoring: how to make the most of a contaminated multivariate sample" by A. Cerioli, M. Riani, A. Atkinson and A. Corbellini that will appear in Statistical Methods & Applications.We would like to congratulate Cerioli, Riani, Atkinson and Corbellini (henceforth CRAC) on their well-written and lavishly illustrated exposition about the usage and benefits of monitoring, and thank the editors for inviting us to comment on this interesting work. The problem of nearby contaminationThe leading example in the paper is the geyser (Old Faithful) dataset. From the scatterplot of this bivariate dataset we see that it consists of two clusters, the smaller of which contains about 30% to 35% of the observations. If one interprets the smaller cluster as contamination this is a relatively high contamination level, though it should not be prohibitive since the estimators considered in the paper can be tuned to a breakdown value well above 35%. However, the contamination happens to lie quite close to the inlying data. Having a large fraction of contamination located fairly close by makes the geyser data particularly challenging, as illustrated by CRAC. If we use a scatter estimator with a breakdown value of e.g. 40%, replacing any 35% of clean data by data points positioned anywhere cannot completely destroy the scatter matrix (in the sense of making its first eigenvalue arbitrarily large or its last eigenvalue arbitrarily close to zero), but that does not imply that the scatter matrix will have a small bias. Indeed, it is known that the bias of the estimators under study is the largest for nearby contamination, as shown by Hubert et al. (2014).The other real data example in the paper is the cows dataset with 4 variables. Our first instinct was to carry out a PCA to get some idea about the shape of the data. Figure 1 shows the first two principal components of the cows data, which explain 96% of the total variance. (Here we used the ROBPCA method of Hubert et al. (2005), but classical PCA gave a very similar picture.) The plot shows that this dataset is equally challenging. CRAC interpret it as a well-behaved point cloud plus contamination. Again most of the contamination is nearby, and then it fans out. An alternative interpretation is that it could arXiv:1803.04820v1 [stat.ME]

show abstract

“…We have ported the MixSim R package to the MATLAB FSDA toolbox of Riani et al (2012), thus providing an easy-to-use unified framework in which data generation, state-of-the art robust clustering algorithms and principled routines for their evaluation are now integrated. Our effort to produce a rich variety of robust and computationally efficient methods for the analysis of complex data has also lead to an improved algorithm for approximating the distribution function of quadratic forms.…”

Section: Conclusion and Next Stepsmentioning

confidence: 99%

“…First of all, we have ported the MixSim R package, including the long and complex C code at the basis of the misclassification probabilities estimation, to the MATLAB FSDA toolbox of Riani et al (2012). Flexible Statistics for Data Analysis (FSDA) is a statistical library providing a rich variety of robust and computationally efficient methods for the analysis of complex data, possibly affected by outliers and other sources of heterogeneity.…”

Section: Introductionmentioning

confidence: 99%

Simulating mixtures of multivariate data with fixed cluster overlap in FSDA library

Riani

Cerioli

Perrotta

et al. 2015

Adv Data Anal Classif

Self Cite

View full text Add to dashboard Cite

We extend the capabilities of MixSim, a framework which is useful for evaluating the performance of clustering algorithms, on the basis of measures of agreement between data partitioning and flexible generation methods for data, outliers and noise. The peculiarity of the method is that data are simulated from normal mixture distributions on the basis of pre-specified synthesis statistics on an overlap measure, defined as a sum of pairwise misclassification probabilities. We provide new tools which enable us to control additional overlapping statistics and departures from homogeneity and sphericity among groups, together with new outlier contamination schemes. The output of this extension is a more flexible framework for generation of data to better address modern robust clustering scenarios in presence of possible contamination. WeThe FSDA project and the related research are partly supported by the University of Parma through the MIUR PRIN "MISURA-Multivariate models for risk assessment" project and by the Joint Research Centre of the European Commission, through its work program 2014-2020. Electronic supplementary materialThe online version of this article (doi:10.1007/s11634-015-0223-9) contains supplementary material, which is available to authorized users. also study the properties and the implications that this new way of simulating clustering data entails in terms of coverage of space, goodness of fit to theoretical distributions, and degree of convergence to nominal values. We demonstrate the new features using our MATLAB implementation that we have integrated in the Flexible Statistics for Data Analysis (FSDA) toolbox for MATLAB. With MixSim, FSDA now integrates in the same environment state of the art robust clustering algorithms and principled routines for their evaluation and calibration. A spin off of our work is a general complex routine, translated from C language to MATLAB, to compute the distribution function of a linear combinations of non central χ 2 random variables which is at the core of MixSim and has its own interest for many test statistics.

show abstract

FSDA: A MATLAB toolbox for robust analysis and interactive data exploration

Cited by 73 publications

References 35 publications

Discussion of “The power of monitoring: how to make the most of a contaminated multivariate sample”

Discussion of “The power of monitoring: how to make the most of a contaminated multivariate sample”

Discussion of “The power of monitoring: how to make the most of a contaminated multivariate sample”

Simulating mixtures of multivariate data with fixed cluster overlap in FSDA library

Contact Info

Product

Resources

About