Enhanced DQN Framework for Selecting Actions and Updating Replay Memory Considering Massive Non-Executable Actions

Abstract

A Deep-Q-Network (DQN) controls a virtual agent as the level of a player using only screenshots as inputs. Replay memory selects a limited number of experience replays according to an arbitrary batch size and updates them using the associated Q-function. Hence, relatively fewer experience replays of different states are utilized when the number of states is fixed and the state of the randomly selected transitions becomes identical or similar. The DQN may not be applicable in some environments where it is necessary to perform the learning process using more experience replays than is required by the limited batch size. In addition, because it is unknown whether each action can be executed, a problem of an increasing amount of repetitive learning occurs as more non-executable actions are selected. In this study, an enhanced DQN framework is proposed to resolve the batch size problem and reduce the learning time of a DQN in an environment with numerous non-executable actions. In the proposed framework, non-executable actions are filtered to reduce the number of selectable actions to identify the optimal action for the current state. The proposed method was validated in Gomoku, a strategy board game, in which the application of a traditional DQN would be difficult.

UX Framework Including Imbalanced UX Dataset Reduction Method for Analyzing Interaction Trends of Agent Systems

“…Jian Zheng et al used partial mutual information (PMI) [ 49 ] and correlation matching-based active learning (CMAL) [ 50 ] to suggest a technique to reduce the amount of training data [ 51 ]. Gu et al suggested a method to reduce the amount of training data by grouping the training data of the Gomoku game in similar states together with an ANN [ 52 , 53 ]. This method significantly reduced the amount of training data and induced the DNN to select the next best solution through the action filter.…”

“…As the UX dataset for each user can be an imbalanced dataset, overfitting may occur when learning without preprocessing. In this paper, the amount of data is reduced by grouping similar UX data within the same class and selecting representative data based on the similarity calculated by ANN [ 52 , 53 ]. Figure 4 shows the flowchart of the ANN-based clustering method.…”