Reliability of internal prediction/estimation and its application. I. Adaptive action selection reflecting reliability of value function

Abstract

This article proposes an adaptive action-selection method for a model-free reinforcement learning system, based on the concept of the 'reliability of internal prediction/estimation'. This concept is realized using an internal variable, called the Reliability Index (RI), which estimates the accuracy of the internal estimator. We define this index for a value function of a temporal difference learning system and substitute it for the temperature parameter of the Boltzmann action-selection rule. Accordingly, the weight of exploratory actions adaptively changes depending on the uncertainty of the prediction. We use this idea for tabular and weighted-sum type value functions. Moreover, we use the RI to adjust the learning coefficient in addition to the temperature parameter, meaning that the reliability becomes a general basis for meta-learning. Numerical experiments were performed to examine the behavior of the proposed method. The RI-based Q-learning system demonstrated its features when the adaptive learning coefficient and large RI-discount rate (which indicate how the RI values of future states are reflected in the RI value of the current state) were introduced. Statistical tests confirmed that the algorithm spent more time exploring in the initial phase of learning, but accelerated learning from the midpoint of learning. It is also shown that the proposed method does not work well with the actor-critic models. The limitations of the proposed method and its relationship to relevant research are discussed. q

Artificial Neural Network Model for Predicting 5-year Mortality after Surgery for Hepatocellular Carcinoma and Performance Comparison with Logistic Regression Model: A Nationwide Taiwan Database Study

“…A literature review shows that this study is the first to use a nationwide population-based database for training and testing a neural network to predict HCC surgery outcomes. Unlike other standard statistical models, ANNs can also manage complexity even with small samples and with an unbalanced ratio between variables and records [8][9][10]. That is, ANNs overcome the problem of dimensionality.…”

“…Although many different ANNs have been developed, one of the most common structures consists of an interconnected group of nodes in multiple layers, in which input nodes and output nodes have clinical correlates [9,10]. The nodes are connected by links, each of which has an associated weight.…”

“…The nodes are connected by links, each of which has an associated weight. This network is "trained" by exposure to inputs paired with known outputs, and learning occurs when the weights between nodes are modified according to feedback [9,10]. The computational power of an ANN is derived from the distributed nature of its connections.…”

“…The computational power of an ANN is derived from the distributed nature of its connections. Once a model is trained, novel records can be used as input to improve its prediction capability [8][9][10].…”

A Deterministic Improved Q-Learning for Path Planning of a Mobile Robot

“…In [38], the authors employed an interesting strategy of internal prediction/estimation to control the balance between exploration and exploitation for efficient adaptability of an agent in a new environment. Here, a reliability parameter RI has been introduced as an internal variable to estimate the 'expected prediction error'.…”