The Epic Story of Maximum Likelihood

Stigler, Stephen M.

doi:10.1214/07-sts249

Cited by 120 publications

(63 citation statements)

References 75 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Section 13.3 in Lehmann and Romano demonstrates local optimality of the score test, and notes the difficulty created by remote alternatives. For the history of likelihood methods, see Stigler (2007). Huber (1967) discusses the behavior of the MLE when the model is wrong-which includes the behavior ofθ S under θ T .…”

Section: Inconsistency Due To Spurious Rootsmentioning

confidence: 99%

How Can the Score Test Be Inconsistent?

Freedman

2007

The American Statistician

View full text Add to dashboard Cite

The score test can be inconsistent because-at the MLE under the null hypothesis-the observed information matrix generates negative variance estimates. The test can also be inconsistent if the expected likelihood equation has spurious roots.KEY WORDS: Maximum likelihood, score test inconsistent, observed information, multiple roots for likelihood equationTo appear in The American Statistician vol. 61 (2007) pp. 291-295 The short answerAfter a sketch of likelihood theory, this paper will answer the question in the title. In brief, suppose we use Rao's score test, normalized by observed information rather than expected information. Furthermore, we compute observed information atθ S , the parameter value maximizing the log likelihood over the null hypothesis. This is a restricted maximum. At a restricted maximum, observed information can generate negative variance estimates-which makes inconsistency possible.At the unrestricted maximum, observed information will typically be positive definite. Thus, if observed information is computed at the unrestricted MLE, consistency should be restored. The "estimated expected" information is also a good option, when it can be obtained in closed form. (Similar considerations apply to the Wald test.) However, the score test may have limited power if the "expected likelihood equation" has spurious roots: details are in section 8 below.The discussion provides some context for the example in Morgan, Palmer, and Ridout (2007), and may clarify some of the inferential issues. Tedious complications are avoided by requiring "suitable regularity conditions" throughout: in essence, densities are positive, smooth functions that decay rapidly at infinity, and with some degree of uniformity. Mathematical depths can be fathomed another day.David A. Freedman is Professor, Department of Statistics, University of California Berkeley, CA 94720-3860 (E-mail: freedman@stat.berkeley.edu). Peter Westfall (Texas Tech) made many helpful comments, as did Morgan, Palmer, and Ridout.

show abstract

Section: Inconsistency Due To Spurious Rootsmentioning

confidence: 99%

How Can the Score Test Be Inconsistent?

Freedman

2007

The American Statistician

View full text Add to dashboard Cite

show abstract

“…As early as in 1930, in his letters to Fisher, Hotelling raised the possibility of the MLE performing poorly [149]. Subsequently, various examples showing that the performance of the MLE can be significantly improved upon have been proposed in the literature, see Le Cam [150] for an excellent overview.…”

Section: Motivation Methodology and Related Workmentioning

confidence: 99%

Minimax Estimation of Functionals of Discrete Distributions

Jiao

Venkat

Han

et al. 2015

IEEE Trans. Inform. Theory

144

View full text Add to dashboard Cite

Abstract-We propose a general methodology for the construction and analysis of essentially minimax estimators for a wide class of functionals of finite dimensional parameters, and elaborate on the case of discrete distributions, where the support size S is unknown and may be comparable with or even much larger than the number of observations n. We treat the respective regions where the functional is "nonsmooth" and "smooth" separately. In the "nonsmooth" regime, we apply an unbiased estimator for the best polynomial approximation of the functional whereas, in the "smooth" regime, we apply a biascorrected version of the Maximum Likelihood Estimator (MLE).We illustrate the merit of this approach by thoroughly analyzing the performance of the resulting schemes for estimating two important information measures: the entropyWe obtain the minimax L2 rates for estimating these functionals. In particular, we demonstrate that our estimator achieves the optimal sample complexity n S/ ln S for entropy estimation. We also demonstrate that the sample complexity for estimating Fα(P ), 0 < α < 1, is n S 1/α / ln S, which can be achieved by our estimator but not the MLE. For 1 < α < 3/2, we show the minimax L2 rate for estimating Fα(P ) is (n ln n) −2(α−1) for infinite support size, while the maximum L2 rate for the MLE is n −2(α−1) . For all the above cases, the behavior of the minimax rate-optimal estimators with n samples is essentially that of the MLE (plug-in rule) with n ln n samples, which we term "effective sample size enlargement".We highlight the practical advantages of our schemes for the estimation of entropy and mutual information. We compare our performance with various existing approaches, and demonstrate that our approach reduces running time and boosts the accuracy. Moreover, we show that the minimax rate-optimal mutual information estimator yielded by our framework leads to significant performance boosts over the Chow-Liu algorithm in learning graphical models. The wide use of information measure estimation suggests that the insights and estimators obtained in this work could be broadly applicable.

show abstract

“…This becomes important when we realise that Spearman (1904) tried but was never able to offer a mathematical proof for the original factor theory (Cowles, 2001). Further, the maximum likelihood technique, typically used to estimate parameters in the latent variable model, remained without mathematical proof despite the best efforts of its original developer, Ronald Fisher (see Stigler, 2007). The latent variable model therefore cannot be described offering deductive closure as a model, given these proof gaps (see Suppes, 1999;Tarski, 1930).…”

Section: Logical Frameworkmentioning

confidence: 99%

Towards a Logical Framework for Latent Variable Modelling

Nowland¹,

Beath²,

Boag³

2018

EasyChair Preprints

View full text Add to dashboard Cite

Modelling in psychometrics has become increasingly reliant on computer software; at the same time, many decisions that a researcher makes remain unrecorded and perhaps, unreconciled to anything more than the researcher's intuition or best guess. The aim of this paper is to set out a logic that accounts for and guides decision procedures, in psychometric research practices. Such a logic is informed by the integration of three systematic viewpoints:

show abstract

The Epic Story of Maximum Likelihood

Cited by 120 publications

References 75 publications

How Can the Score Test Be Inconsistent?

How Can the Score Test Be Inconsistent?

Minimax Estimation of Functionals of Discrete Distributions

Towards a Logical Framework for Latent Variable Modelling

Contact Info

Product

Resources

About