A critical look at experimental evaluations of EBL

Abstract

Abstract. A number of experimental evaluations of explanation-based learning (EBL) have been reported in the literature on machine learning. A close examination of the design of these experiments reveals certain methodological problems that could affect the conclusions drawn from the experiments. This article analyzes some of the more common methodological difficulties, and illustrates them using selected previous studies.

Statistical methods for analyzing speedup learning experiments

“…We applied both tests to the speedup learning data set taken from Etzioni (1990a) and have shown that most of the differences observed are statistically significant (see, in particular, the results of our extended signed rank test in table 6). We believe that this approach helps to allay the concerns regarding the use of resource bounds raised by Segre et al (1991). Finally, although we have focused on speedup learning data, we note that our methodology can be used to analyze any quantitative comparison between two systems on a common set of problems.…”

“…In a recent paper, Segre et al (1991) argue that the experimenter's choice of time bound can influence the results of the experiment. Segre et al illustrate this point with a hypothetical example reproduced in tables 1 and 2.…”

“…Often, the experimenter imposes a bound on the CPU time the problem solver is allowed to spend on any individual problem. Segre et al (1991) argue that the experimenter's choice of time bound can bias the results of the experiment. To address this problem, we present statistical hypothesis tests specifically designed to analyze speedup data and eliminate this bias.…”

Automatically generating abstractions for planning

“…Similar comparisons in the past have been done using cumulative time graphs [45], but Segre et al [55] argue that such comparisons could be misleading because changing the time limit can change the results. To avoid this problem, the total time expended solving all of the problems is graphed against the CPU time bound.…”

On-line learning from search failures

“…We used the number of problems solved within the assigned resource limit as an additional measure of performance. As Segre et al (1991) point out, comparisons based on these measures may be sensitive to the values of the resource limits, and inconclusive when some of the problems are not solved within the assigned resource limits. Figure 4 shows FAILSAFE-2's performance in the blocks world domain.…”