An analysis of optimistic, best-first search for minimax sequential decision making

Abstract

Abstract-We consider problems in which a maximizer and a minimizer agent take actions in turn, such as games or optimal control with uncertainty modeled as an opponent. We extend the ideas of optimistic optimization to this setting, obtaining a search algorithm that has been previously considered as the best-first search variant of the B* method. We provide a novel analysis of the algorithm relying on a certain structure for the values of action sequences, under which earlier actions are more important than later ones. An asymptotic branching factor is defined as a measure of problem complexity, and it is used to characterize the relationship between computation invested and near-optimality. In particular, when action importance decreases exponentially, convergence rates are obtained. Throughout, examples illustrate analytical concepts such as the branching factor. In an empirical study, we compare the optimistic best-first algorithm with two classical game tree search methods, and apply it to a challenging HIV infection control problem.

Near-optimal control of nonlinear switched systems with non-cooperative switching rules

“…We extend the analysis of OMS in [5] to OMSδ. The first part of our analysis establishes basic properties of the minimax algorithm that still hold under the additional dwell-time constraints.…”

“…The second part gives our main novel results: a complexity measure of the problem and a corresponding convergence rate of OMSδ. Due to space limits we skip all proofs except that of the main result, but where applicable we point out relations to [5].…”

“…At each iteration, to choose the next leaf to expand, OMSδ starts from the root and constructs a path by recursively selecting an optimistic child for the agent at the current node, in the same way as OMS in [5]: a child with the largest upper bound at max nodes, and one with the smallest lower bound at min nodes. The main difference between OMSδ and OMS is in the expansion of this leaf, which in OMSδ is constrained to only create the children that obey the dwell time conditions, as explained above.…”

“…The modes can have arbitrary nonlinear dynamics, while the rewards must be bounded. This is a minimax problem, which we solve by extending the approach from [5], called optimistic minimax search (OMS). OMS explores a tree representation of the possible sequences of max and min agent actions (here, mode switches); it is a variant of B* [17] and related to other classical minimax search methods [10], [18], [12].…”

“…In the context of artificial intelligence and optimistic planning [16], [11], [9], [22], [15], the closest algorithm is again OMS [5], compared to which the main novelty here is the convergence analysis under dwell-time constraints, leading to a new complexity measure. The planning method for max-only switched systems from our work [3] leads to a similar complexity measure and reduction compared to the no-dwell-time case, but there the analysis is much easier due to the lack of the min agent.…”