Multi-Stage Temporal Difference Learning for 2048

Wu, I‐Chen; Yeh, Kun Hao; Liang, Chao Chin; Chang, Chia Chuan; Chiang, Han

doi:10.1007/978-3-319-13987-6_34

Cited by 12 publications

(7 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that this technique is different from the one introduced for 2048 by Wu et al [38]. Their algorithm learns the n-tuple networks stage by stage.…”

Section: E Carousel Shaping and Redundant Encodingmentioning

confidence: 95%

“…Their best function involved nearly 23 million parameters and scored 100 178 on average at 1-ply. Wu et al [38] has later extended this work by employing TD(0) with 5 million training games to learn a larger systematic n-tuple system with 67 million parameters, scoring 142 727 on average. By employing three such ntuple networks enabled at different stages of the game along with expectimax with depth 5, they were able to achieve 328 946 points on average.…”

Section: Related Workmentioning

confidence: 99%

“…Following Wu et al [38], as our baseline function approximator, we use 33-42 network shown in Fig. 3c.…”

Section: B Function Approximationmentioning

confidence: 99%

“…To further improve the learning process, we split the game into stages and used a separate function approximator for each game stage. This method has been already used in Buro's Logistello [6], a master-level Othello playing program, and its variant with manually selected stages has already been successfully applied to 2048 [38]. The motivation for multiple stages for 2048 results from the observation that in order to precisely approximate the statevalue function, certain features should have different weights depending on the stage of the game.…”

Section: Multi-stage Weight Promotion 1) Stagesmentioning

confidence: 99%

See 3 more Smart Citations

Mastering 2048 With Delayed Temporal Coherence Learning, Multistage Weight Promotion, Redundant Encoding, and Carousel Shaping

Jaśkowski

2018

IEEE Trans. Games

View full text Add to dashboard Cite

Abstract-2048 is an engaging single-player nondeterministic video puzzle game, which, thanks to the simple rules and hard-to-master gameplay, has gained massive popularity in recent years. As 2048 can be conveniently embedded into the discrete-state Markov decision processes framework, we treat it as a testbed for evaluating existing and new methods in reinforcement learning. With the aim to develop a strong 2048 playing program, we employ temporal difference learning with systematic n-tuple networks. We show that this basic method can be significantly improved with temporal coherence learning, multi-stage function approximator with weight promotion, carousel shaping, and redundant encoding. In addition, we demonstrate how to take advantage of the characteristics of the n-tuple network, to improve the algorithmic effectiveness of the learning process by delaying the (decayed) update and applying lock-free optimistic parallelism to effortlessly make advantage of multiple CPU cores. This way, we were able to develop the best known 2048 playing program to date, which confirms the effectiveness of the introduced methods for discrete-state Markov decision problems.

show abstract

“…Note that this technique is different from the one introduced for 2048 by Wu et al [38]. Their algorithm learns the n-tuple networks stage by stage.…”

Section: E Carousel Shaping and Redundant Encodingmentioning

confidence: 95%

Section: Related Workmentioning

confidence: 99%

“…Following Wu et al [38], as our baseline function approximator, we use 33-42 network shown in Fig. 3c.…”

Section: B Function Approximationmentioning

confidence: 99%

Section: Multi-stage Weight Promotion 1) Stagesmentioning

confidence: 99%

See 2 more Smart Citations

Mastering 2048 With Delayed Temporal Coherence Learning, Multistage Weight Promotion, Redundant Encoding, and Carousel Shaping

Jaśkowski

2018

IEEE Trans. Games

View full text Add to dashboard Cite

show abstract

“…Note that the preliminary version of this paper [32] did not include the following: MS-TD applied to Threes, more splitting strategies for both 2048 and Threes, and a demonstration of combining MS-TD with other techniques to improve 2048 and Threes programs.…”

Section: Introductionmentioning

confidence: 99%