On a Minimum Distance Procedure for Threshold Selection in Tail Analysis

Abstract

Power-law distributions have been widely observed in different areas of scientific research. Practical estimation issues include how to select a threshold above which observations follow a power-law distribution and then how to estimate the power-law tail index. A minimum distance selection procedure (MDSP) is proposed in [8] and has been widely adopted in practice, especially in the analyses of social networks. However, theoretical justifications for this selection procedure remain scant. In this paper, we study the asymptotic behavior of the selected threshold and the corresponding power-law index given by the MDSP. We find that the MDSP tends to choose too high a threshold level and leads to Hill estimates with large variances and root mean squared errors for simulated data with Pareto-like tails.

Scale-free networks well done

“…10 and 11, respectively. Figure 10 illustrates that the KS distance minimization component of the PLFit drives the values of k min that the PLFit attempts to estimate to erroneously low values, in full agreement with more recent and in-depth investigations in [24]. This happens because the smaller the k min , the smaller the deviations of the empirical CDF at degrees k right above k min from the theoretical CDF, because if the distribution is regularly varying, there are more nodes with smaller degrees.…”

“…Remarkably, one example of such a distribution is the degree distribution in the preferential attachment model, a "harmonic oscillator" of power laws in network science [28][29][30]. For these and a number of other lower-level technical reasons, all documented in Appendix D and fully supported in a more recent and detailed focused study [24], we exclude the PLFit from the subsequent considerations here.…”

“…The algorithm is thus a mixture of two optimization strategies: one is based on likelihood maximization, while the other one deals with the KS distance minimization. We note that since the algorithm does not implement MLE exactly, it trivially cannot be consistent if the true value of γ does not belong to the finite set of γ s values, because it can never report any γ-estimateγ that does not belong to the finite set of γ s s. More importantly, even though the correct implementation of MLE with a fixed and known k min had long been proven to be consistent [19], the consistency of MLE in combination with KS-distance minimization has been proven only very recently in [24], and only for pure power laws, i.e., for the Pareto or generalized zeta distributions. If the distribution is not a pure power law but a general regularly varying distribution, the consistency of the algorithm is a question that has not been rigorously explored at all, except the conjectures in [24] that this MLE-KS combination appears to be consistent for regularly varying distributions satisfying the second order condition, and for regularly varying distributions whose slowly varying functions (k) converge to constants, and that the algorithm is likely not to be consistent for all other classes of regularly varying distributions.…”

“…Yet the distribution of such ks may converge to a pure power law, but only if the k min value that the algorithm finds diverges with the sample size n, and only if the slowly varying function (k) converges to a constant. Therefore, for this subclass of regularly varying distributions with (k)s converging to constants, the algorithm is likely to be consistent, yet the full proof is currently lacking [24].…”

“…Specifically, it is quite unclear what it really means for a degree sequence in a given real-world network to be power-law or "close" to a power law. This lack of rigor has led and still leads to confused controversy and never-ending heated debates [9][10][11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26][27]. This controversy has culminated in the recent work [20] that concluded that "scale-free networks are rare."…”