Developing a 2048 Player with Backward Temporal Coherence Learning and Restart

Playing Game 2048 with Deep Convolutional Neural Networks Trained by Supervised Learning

“…Several computer players have been developed for Game 2048. The most widely used and successful approach is based on Ntuple networks (NTNs) trained by reinforcement learning methods [11], [15], [25], [29]. Neural networks (NN) are also popularly used in the development of Game 2048 computer players [1], [6], [9], [21], [26], [27], [28].…”

“…The feature weights are adjusted by reinforcement learning methods. For Game 2048 players, the temporal difference learning (TD learning) was commonly used [25], [29], [32], and then a learning-rate-free variant (temporal coherence learning; TC learning) was introduced [11], [15]. Due to the characteristics of the game, biasing the training actions to learn sometimes improves the performance such as carousel shaping [11] and restart strategy [15].…”

“…Since the weights are basically independent from each other, NTNs do not obtain some important property of the game (for example, the similarity among 1024-2048-4096 tiles and 2048-4096-8192 tiles). Weight promotion [11], [15], which initializes a first-accessed weight with a certain existing one, can be considered as a human-aided solution to this issue. A more affirmative reuse of feature weights achieved even a 65536-tile [10].…”

“…In this paper, we try to improve the performance of DCNNbased players by increasing the number of convolution layers. We designed DCNNs with 2-9 convolution layers and applied supervised learning with play records from existing strong players [15]. As the result, we achieved better results than existing NN-based players.…”

“…We also tested supervised learning of value networks (both an NTN and a smaller variant of tjwei's network) with the same set of training data, but we failed to obtain good players (Section 6). [25] 17×4-tuples, TD learning 860,625 51,320 Szubert and Jaśkowski [25] 2×4-tuples & 2×6-tuples, TD learning 22,882,500 99,916 Wu et al [29], [32] 4×6 -tuples, TD learning, 3 stages 136,687,500 143,958 N-tuple Oka and Matsuzaki [20] 40×6-tuples, TD learning 671,088,640 210,476 network Oka and Matsuzaki [20] 10×7-tuples, TD learning 2,684,354,560 234,136 Matsuzaki [14] 8×6-tuples, TD learning 134,217,728 226,958 Matsuzaki [14] 8×7-tuples, TD learning 2,147,483,648 255,198 Matsuzaki [15] 4×6-tuples, backward TC learning, 8 stages 536,870,912 232,262 Jaśkowski [11] 5×6-tuples, TC learning, 16 stages, redundant encoding, etc. 1,347,551,232 324,710 This work (comparison) 5×4-tuples, TC learning, 3 stages 983,040 50,120 Guei et al [9] 2 convolution (2 × 2), 2 full-connect, TD learning N/A ≈11,400 Guei et al [9] 3 convolution (3 × 3), 2 full-connect, TD learning N/A ≈ 5,300 tjwei [26] 2 The rest of the paper is organized as follows.…”