A Critical Assessment of Benchmark Comparison in Planning

Abstract

Recent trends in planning research h a v e led to empirical comparison becoming commonplace. The eld has started to settle into a methodology for such comparisons, which for obvious practical reasons requires running a subset of planners on a subset of problems. In this paper, we characterize the methodology and examine eight implicit assumptions about the problems, planners and metrics used in many of these comparisons. The problem assumptions are: PR1 the performance of a general purpose planner should not be penalized biased if executed on a sampling of problems and domains, PR2 minor syntactic di erences in representation do not a ect performance, and PR3 problems should be solvable by STRIPS capable planners unless they require ADL. The planner assumptions are: PL1 the latest version of a planner is the best one to use, PL2 default parameter settings approximate good performance, and PL3 time cut-o s do not unduly bias outcome. The metrics assumptions are: M1 performance degrades similarly for each planner when run on degraded runtime environments e.g., machine platform and M2 the number of plan steps distinguishes performance. We nd that most of these assumptions are not supported empirically; in particular, that planners are a ected di erently by these assumptions. We conclude with a call to the community to devote research resources to improving the state of the practice and especially to enhancing the available benchmark problems.

Portfolio-based planning: State of the art, common practice and open challenges

“…This step could be redundant in some portfolio structures: n-of-n design does not require any selection. The configured portfolio includes all the incorporated planners, and is based on the hypothesis that typically planners either solve a problem quickly or not at all [25]. This strategy is reasonable when all the following hold:…”

“…Reasonably, this means that each planner should run for at least a few tens of seconds: this is because, according to [25], if a planner does not find a solution quickly, it will not find it at all. Finally, we would emphasise that if the target of the portfolio is minimising the runtime, including all the incorporated planners will possibly make it hard to effectively order them.…”

“…However, while several powerful domain-independent planners have been developed, none of them clearly outperforms all others in every known benchmark domain. It has also been observed that if a planner does not find a solution quickly, it is very likely it will not find it at all, even with very large CPU time horizons [25]. These observations motivate the idea of configuring and exploiting a portfolio of planners to achieve better overall performance than any individual planner.…”

A Guide to Portfolio-Based Planning

“…This step could be useless in some portfolio structures: n-of-n (ArvandHerd and one of the configuration strategies included in FDSS2) design does not require any selection. The configured portfolio includes all the incorporated planners, and is based on the hypothesis that typical planners either solve a problem quickly or not at all [11]. This strategy is reasonable when: (i) the number of incorporated planners is limited; (ii) all the incorporated planners have really good mean performances; (iii) the maximum amount of CPU time for solving a problem is quite large and, (iv) the target of the portfolio is not minimizing the runtime.…”

Machine Learning for Adaptive Planning

“…One of the few past approaches towards the direction of adaptive planning is the BUS system (Howe & Dahlman, 1993;Howe et al, 1999). BUS runs six state-of-the-art planners, namely STAN, IPP, SGP, BlackBox, UCPOP and PRODIGY, using a round-robin schema, until one of them finds a solution.…”