Consistent Belief State Estimation, with Application to Mines

Abstract

Abstract-Estimating the belief state is the main issue in games with Partial Observation. It is commonly done by heuristic methods, with no mathematical guarantee. We here focus on mathematically consistent belief state estimation methods, in the case of one-player games. We clearly separate the search algorithm (which might be e.g. alpha-beta or Monte-Carlo Tree Search) and the belief state estimation. We basically propose rejection methods and simple Monte-Carlo Markov Chain methods, with a time budget proportional to the time spent by the search algorithm on the situation at which the belief state is to be estimated; this is conveniently approximated by the number of simulations in the current node. While the approach is intended to be generic, we perform experiments on the wellknown Mines game, available on most Windows and Linux distributions. Interestingly, it detects non-trivial facts, e.g. the fact that the probability of winning the game is not the same for different moves, even those with the same probability of immediate death. The rejection method, which is slow but has no parameter and which is consistent in a non-asymptotic setting, performed better than the MCMC method in spite of tuning efforts.

Optimistic Heuristics for MineSweeper

“…considering also moves with non minimal probability of mine) can bring an improvement; but we have no proof. Also we have no proof that always playing in the corner for the first move is a good idea; it is known as a good heuristic, but maybe in some cases there are better moves (with the rule "first move is a 0", it is mathematically proved that the center is a good move in some cases; see [3]). Fig.…”

“…The most classical approach for MineSweeper is based on Constraint Satisfaction Problems [14], as in Section 2.2: this provides a provably correct estimate of the belief state, and then classically after CSP one plays the covered location with least probability of being a mine. However, [3] has shown that this approach is suboptimal; there are situations, as in Fig. 2, where some moves with minimum probability of mines are nonetheless suboptimal moves.…”

“…An alternate approach has been investigated in [3,13]; using UCT, these approaches have asymptotic optimality, and so outperform CSP when given enough time. They got best performance so far on small boards, outperforming CSP, but are too slow for big boards.…”

“…It is widely reported that choosing a heuristic with high performance does not necessarily lead to high performance of the UCT built on top of it [5]. However, in the one player case, introducing strong heuristics is seemingly quite efficient [3,13]. This is the principle of [13], using a strong heuristic, namely CSP, for improving UCT performance on MineSweeper on small boards.…”

“…This heuristic "tie breaking" provides significant improvements. However, as shown by [3], there are cases where the optimal move is difficult to find among moves with minimum probability of mine. Therefore, CSP variants do not have the asymptotic optimality property discussed in the introduction.…”