A general bootstrap performance diagnostic

Abstract

As datasets become larger, more complex, and more available to diverse groups of analysts, it would be quite useful to be able to automatically and generically assess the quality of estimates, much as we are able to automatically train and evaluate predictive models such as classifiers. However, despite the fundamental importance of estimator quality assessment in data analysis, this task has eluded highly automatic solutions. While the bootstrap provides perhaps the most promising step in this direction, its level of automation is limited by the difficulty of evaluating its finite sample performance and even its asymptotic consistency. Thus, we present here a general diagnostic procedure which directly and automatically evaluates the accuracy of the bootstrap's outputs, determining whether or not the bootstrap is performing satisfactorily when applied to a given dataset and estimator. We show that our proposed diagnostic is effective via an extensive empirical evaluation on a variety of estimators and simulated and real datasets, including a real-world query workload from Conviva, Inc. involving 1.7TB of data (i.e., approximately 0.5 billion data points).

Knowing when you're wrong

“…We call such a procedure a diagnostic. We are going to use a diagnostic recently developed by Kleiner et al [22], but first let us provide some intuition for how a diagnostic should work. We consider first an impractical ideal diagnostic.…”

“…Algorithm 1 is the diagnostic of Kleiner et al [22], specialized to query approximation. We provide it here for completeness.…”

“…The past decade has seen several investigations of such diagnostic methods in the statistics literature [12,22]. Recently, Kleiner et al [22] designed a diagnostic algorithm to detect when bootstrapbased error estimates are unreliable.…”

“…Recently, Kleiner et al [22] designed a diagnostic algorithm to detect when bootstrapbased error estimates are unreliable. However, their algorithm was not designed with computational considerations in mind and in-volves tens of thousands of test query executions, making it prohibitively slow in practice.…”

The analytical bootstrap

“…This derivation is a manual process that is ad-hoc and often impractical for complex queries [34]. 1 To address this problem, a second approach, called bootstrap, has emerged as a more general method for routine estimation of errors [27,30,34]. Bootstrap [19] is essentially a Monte Carlo procedure, which for a given initial sample, (i) repeatedly forms simulated datasets by resampling tuples i.i.d.…”