Data-adaptive trimming of the Hill estimator and detection of outliers in the extremes of heavy-tailed data

Abstract

We introduce a trimmed version of the Hill estimator for the index of a heavy-tailed distribution, which is robust to perturbations in the extreme order statistics. In the ideal Pareto setting, the estimator is essentially finite-sample efficient among all unbiased estimators with a given strict upper break-down point. For general heavy-tailed models, we establish the asymptotic normality of the estimator under second order regular variation conditions and also show it is minimax rateoptimal in the Hall class of distributions. We also develop an automatic, data-driven method for the choice of the trimming parameter which yields a new type of robust estimator that can adapt to the unknown level of contamination in the extremes. This adaptive robustness property makes our estimator particularly appealing and superior to other robust estimators in the setting where the extremes of the data are contaminated. As an important application of the data-driven selection of the trimming parameters, we obtain a methodology for the principled identification of extreme outliers in heavy tailed data. Indeed, the method has been shown to correctly identify the number of outliers in the previously explored Condroz data set.

Threshold selection and trimming in extremes

“…exponential sample with mean ξ . Bhattacharya et al (2019) recently proposed linear estimators of the form…”

Robust Estimation of Loss Models for Truncated and Censored Severity Data

“…T -estimators have been discussed in the operational risk literature by Opdyke and Cavallo (2012), used in credibility studies by Kim and Jeon (2013), and further tested in risk measurement exercises by Abu Bakar and Nadarajah (2019). Also, the idea of trimming has been gaining popularity in modeling extremes (see Bhattacharya et al, 2019;Bladt et al, 2020). Thus we anticipate the methodology developed in this paper will be useful and transferable to all these and other areas of research.…”

“…To get point estimates Π[ F ],we plug in the estimates of α from Table5.2 into (5.2). To construct interval estimators, we rely on the delta method (seeSerfling, 1980, Section 3.3), which uses the asymptotic distributions(4…”

“…4. Comparison of W 's and censored MLEs of α of Pareto I (d = 500, 000, α) fitted to the original and modified data sets.…”

Outlier detection and a tail-adjusted boxplot based on extreme value theory

“…This allows for detecting extreme outliers or sets of extreme data that show less spread than the bulk of the data. To this end we extend a testing method proposed in Bhattacharya et al (2019) for the specific case of heavy tailed models, to all max-domains of attraction. Consequently we propose a tail-adjusted boxplot which yields a more accurate representation of possible outliers.…”