Semi-analytical method for analyzing models and model selection measures based on moment analysis

Dhurandhar, Amit; Dobra, Alin

doi:10.1145/1497577.1497579

Cited by 7 publications

(6 citation statements)

References 21 publications

(18 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…observations (e.g. [3,4,6,14,15,22,23,30,32,33,37,39]). However, none of them address the problems of (1) dependence between related instances and (2) dependence between training and test set sizes.…”

Section: Both Of the Above Methodologies Generally Address The Statismentioning

confidence: 99%

Correcting evaluation bias of relational classifiers with network cross validation

et al. 2011

View full text Add to dashboard Cite

Recently, a number of modeling techniques have been developed for data mining and machine learning in relational and network domains where the instances are not independent and identically distributed (i.i.d.). These methods specifically exploit the statistical dependencies among instances in order to improve classification accuracy. However, there has been little focus on how these same dependencies affect our ability to draw accurate conclusions about the performance of the models. More specifically, the complex link structure and attribute dependencies in relational data violate the assumptions of many conventional statistical tests and make it difficult to use these tests to assess the models in an unbiased manner. In this work, we examine the task of within-network classification and the question of whether two algorithms will learn models that will result in significantly different levels of performance. We show that the commonly used form of evaluation (paired t-test on overlapping network samples) can result in an unacceptable level of Type I error. Furthermore, we show that Type I error increases as (1) the correlation among instances increases and (2) the size of the evaluation set increases (i.e., the proportion of labeled nodes in the network decreases). We propose a method for network cross-validation that combined with paired t-tests produces more acceptable levels of Type I error while still providing reasonable levels of statistical power (i.e., 1−Type II error).

show abstract

Section: Both Of the Above Methodologies Generally Address The Statismentioning

confidence: 99%

Correcting evaluation bias of relational classifiers with network cross validation

et al. 2011

View full text Add to dashboard Cite

show abstract

“…We then revisit the generic expressions provided in [Dhurandhar and Dobra 2009] that need to be customized to specific classification algorithms; in this case an ensemble of random decision trees. Finally, we provide the customized expressions for random decision trees (not ensemble) of prespecified height and discuss the intuitions in its derivation.…”

Section: Preliminariesmentioning

confidence: 99%

“…Hence, having fewer variables will invariably lead to tighter bounds. If we were to derive bounds directly, we would have to consider the entire dataset with all the cells without collapsing or aggregation of cells into fewer cells and hence fewer variables [Dhurandhar and Dobra 2009], as is the case using the generic expressions. The reason for this is that the generic expressions consider cells corresponding to an individual input or pairs of inputs rather than the entire input space.…”

Section: Practical Considerationsmentioning

confidence: 99%

“…In previous works [Dhurandhar and Dobra 2009], the authors provided a moment based methodology to accurately characterize classification models. In particular, they provided generic expressions to compute the first two moments of the generalization error -expected error over the entire input, given an underlying probability distribution.…”

Section: Introductionmentioning

confidence: 99%

“…The challenge was then to customize these expressions to specific algorithms in order to meticulously study them. In [Dhurandhar and Dobra 2009;2008b;2008a], these expressions were customized for the Naive Bayes Classifier, Nearest Neighbor Classifier and (single) Random decision trees (not an ensemble). It was seen that such expressions have the potential to provide extremely tight characterizations of the behavior of individual classification algorithms as well as certain model selection techniques (viz.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Bounds on the moments for an ensemble of random decision trees

Dhurandhar¹

2014

Knowl Inf Syst

Self Cite

View full text Add to dashboard Cite

An ensemble of random decision trees is a popular classification technique, especially known for its ability to scale to large domains. In this paper, we provide an efficient strategy to compute bounds on the moments of the generalization error computed over all datasets of a particular size drawn from an underlying distribution, for this classification technique. Being able to estimate these moments can help us gain insights into the performance of this model. As we will see in the experimental section these bounds tend to be significantly tighter than the state-of-the-art Breiman's bounds based on strength and correlation and hence, more useful in practice.

show abstract

On the relationship between the discrete and continuous bounding moment problems and their numerical solutions

2016

View full text Add to dashboard Cite

We present a brief survey of some of the basic results related to the classical continuous moment problems (CMP) and the recently developed discrete moment problems (DMP), clarifying their relationship and propose new methods for the solution of univariate continuous and discrete power moment problems. In the classical as well as in the recently developed discrete moment problems the coefficient function in the objective is supposed to be higher order convex (in the entire interval or part of it), or constant in an interval while zero elsewhere, or equal to a constant at some point and zero elsewhere. The concept of a regenerative block (of points) is introduced, for the case of the DMP, that makes it possible to create lower and upper bounds for other functions in the objective. The CMP are solved by discretization and sequential application of dual type algorithm. Numerical results are presented with moments of order up to 40 and various applications are mentioned. KeywordsMoment problems • Semi-infinite linear programs • Discrete moment problems • Bounding probabilities and expectations B András Prékopa

show abstract

Semi-analytical method for analyzing models and model selection measures based on moment analysis

Cited by 7 publications

References 21 publications

Correcting evaluation bias of relational classifiers with network cross validation

Correcting evaluation bias of relational classifiers with network cross validation

Bounds on the moments for an ensemble of random decision trees

On the relationship between the discrete and continuous bounding moment problems and their numerical solutions

Contact Info

Product

Resources

About