Coevolutionary Temporal Difference Learning for Othello

Abstract

This paper presents Coevolutionary Temporal Difference Learning (CTDL), a novel way of hybridizing coevolutionary search with reinforcement learning that works by interlacing one-population competitive coevolution with temporal difference learning. The coevolutionary part of the algorithm provides for exploration of the solution space, while the temporal difference learning performs its exploitation by local search. We apply CTDL to the board game of Othello, using weighted piece counter for representing players' strategies. The results of an extensive computational experiment demonstrate CTDL's superiority when compared to coevolution and reinforcement learning alone, particularly when coevolution maintains an archive to provide historical progress. The paper investigates the role of the relative intensity of coevolutionary search and temporal difference search, which turns out to be an essential parameter. The formulation of CTDL leads also to the introduction of Lamarckian form of coevolution, which we discuss in detail.

Evolving small-board Go players using coevolutionary temporal difference learning with archives

“…To quote Arthur Lee Samuel's declaration, The temptation to improve the machine's game by giving it standard openings or other man-generated knowledge of playing techniques has been consistently resisted (Samuel, 1959, p. 215). This result confirms our former observations (Szubert et al, 2009), when we demonstrated that hybridizing coevolution with TD(0) proves beneficial when learning the strategy of the game of Othello. Here, we come to similar conclusions for the game of small-board Go, and additionally note that extending the lookahead horizon by using TD(λ) with λ close to 1 can boost the performance of CTDL even further.…”

“…4) speeds up the learning, the difference, initially substantial, becomes rather negligible after several hundreds of thousands of training games. Based on these results, we conclude that CTDL+HoF is moderately sensitive to the TDL-CEL ratio and recommend values greater than 8 for this parameter, which confirms our earlier findings for Othello (Szubert et al, 2009).…”

“…We reported our first results with CTDL in the work by Szubert et al (2009), where it was applied to learn strategies of the game of Othello. The overall conclusion was positive for CDTL, which produced strategies that, on average, defeated those learned by TDL and CEL.…”

“…For experiments, five methods were prepared, each being a combination of techniques described in the previous section: CEL, TDL and HoF. Wherever possible, parameters taken directly from our previous comparison of the same set of methods (Szubert et al, 2009) were used. Detailed settings follow.…”

Parallel Alpha-Beta Algorithm on the GPU

“…The heuristics of the tested implementations were the same, incorporating the weighted piece counter and mobility metric as components of the evaluation function [5]. The weight matrix used for piece evalution was the one obtained with coevolution by Szubert [20]. The same weight matrix was also used in move ordering to select the first move in the PV-nodes on the CPU.…”