An Information Measure for Classification

Wallace, Chris S.; Boulton, D. M.

doi:10.1093/comjnl/11.2.185

Cited by 817 publications

(530 citation statements)

References 5 publications

Supporting

Mentioning

527

Contrasting

Unclassified

Order By: Relevance

“…The Minimum Message Length (MML) Principle (Wallace & Boulton, 1968;Comley & Dowe, 2005;Wallace, 2005) is a method for inductive inference that is both Bayesian and compression-based. The similarities and differences with MDL are subtle; see, for example, Section 10.2 of Wallace (2005) or Section 17.4 of Grünwald (2007), or (Wallace & Dowe, 1999a,b).…”

Section: What About Mml?mentioning

confidence: 99%

Suboptimal behavior of Bayes and MDL in classification under misspecification

Grünwald

Langford

2007

Mach Learn

View full text Add to dashboard Cite

We show that forms of Bayesian and MDL inference that are often applied to classification problems can be inconsistent. This means that there exists a learning problem such that for all amounts of data the generalization errors of the MDL classifier and the Bayes classifier relative to the Bayesian posterior both remain bounded away from the smallest achievable generalization error. From a Bayesian point of view, the result can be reinterpreted as saying that Bayesian inference can be inconsistent under misspecification, even for countably infinite models. We extensively discuss the result from both a Bayesian and an MDL perspective.

show abstract

Section: What About Mml?mentioning

confidence: 99%

Suboptimal behavior of Bayes and MDL in classification under misspecification

Grünwald

Langford

2007

Mach Learn

View full text Add to dashboard Cite

show abstract

“…The fundamental idea is that compact coding theory provides the right framework in which to think about inference and prediction (Wallace and Boulton [1968]; Wallace and Freeman [1987]; Wallace and Dowe [1999a]; Wallace [2005] (Wallace and Freeman [1987]) is derived from a quadratic Taylor series approximation of Strict MML, and shares many of the desirable features of Strict MML.…”

Section: The Minimum Message Length (Mml) Principlementioning

confidence: 99%

Bayes not Bust! Why Simplicity is no Problem for Bayesians

Dowe

Gardner

Oppy

2007

The British Journal for the Philosophy of Science

View full text Add to dashboard Cite

The advent of formal definitions of the simplicity of a theory has important implications for model selection. But what is the best way to define simplicity?Forster and Sober ([1994]) advocate the use of Akaike's Information Criterion (AIC), a non-Bayesian formalisation of the notion of simplicity. This forms an important part of their wider attack on Bayesianism in the philosophy of science. We defend a Bayesian alternative: the simplicity of a theory is to be characterised in terms of Wallace's Minimum Message Length (MML). We show that AIC is inadequate for many statistical problems where MML performs well. Whereas MML is always defined, AIC can be undefined. Whereas MML is not known ever to be statistically inconsistent, AIC can be. Even when defined and consistent, AIC performs worse than MML on small sample sizes. MML is statistically invariant under 1-to-1 reparametrisation, thus avoiding a common criticism of Bayesian approaches. We also show that MML provides answers to many of Forster's objections to Bayesianism.Hence an important part of the attack on Bayesianism fails.

show abstract

“…The clustering is performed in advance of the prediction process by the clustering program Snob [3], using the content lemmas (unigrams) in the answers as features. The predictive model is a Decision Graph [4] trained on (1) input features: lemma unigrams and bigrams in the request, 3 and (2) target feature -the identifier of the answer cluster that contains the actual answer for the request.…”

Section: Predict a Complete Answermentioning

confidence: 99%

“…The predictive model is a Decision Graph [4] trained on (1) input features: lemma unigrams and bigrams in the request, 3 and (2) target feature -the identifier of the answer cluster that contains the actual answer for the request. The model provides a prediction of which response cluster is most suitable for a given request, as well as a level of confidence in this prediction.…”

Section: Predict a Complete Answermentioning

confidence: 99%

A Comparative Study of Information-Gathering Approaches for Answering Help-Desk Email Inquiries

Zukerman

Marom

2006

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. We present a comparative study of corpus-based methods for the automatic synthesis of email responses to help-desk requests. Our methods were developed by considering two operational dimensions: (1) information-gathering technique, and (2) granularity of the information. In particular, we investigate two techniques -retrieval and prediction -applied to information represented at two levels of granularity -sentence level and document level. We also developed a hybrid method that combines prediction with retrieval. Our results show that the different approaches are applicable in different situations, addressing a combined 72% of the requests with either complete or partial responses.

show abstract

An Information Measure for Classification

Cited by 817 publications

References 5 publications

Suboptimal behavior of Bayes and MDL in classification under misspecification

Suboptimal behavior of Bayes and MDL in classification under misspecification

Bayes not Bust! Why Simplicity is no Problem for Bayesians

A Comparative Study of Information-Gathering Approaches for Answering Help-Desk Email Inquiries

Contact Info

Product

Resources

About