Graph kernels and Gaussian processes for relational reinforcement learning

Abstract

RRL is a relational reinforcement learning system based on Q-learning in relational state-action spaces. It aims to enable agents to learn how to act in an environment that has no natural representation as a tuple of constants. For relational reinforcement learning, the learning algorithm used to approximate the mapping between state-action pairs and their so called Q(uality)-value has to be very reliable, and it has to be able to handle the relational representation of state-action pairs. In this paper we investigate the use of Gaussian processes to approximate the Q-values of state-action pairs. In order to employ Gaussian processes in a relational setting we propose graph kernels as a covariance function between state-action pairs. The standard prediction mechanism for Gaussian processes requires a matrix inversion which can become unstable when the kernel matrix has low rank. These instabilities can be avoided by employing QR-factorization. This leads to better and more stable performance of the algorithm and a more efficient incremental update mechanism. Experiments conducted in the blocks world and with the Tetris game show that Gaussian processes with graph kernels can compete with, and often improve on, regression trees and instance based regression as a generalization algorithm for RRL.

An Autonomic Testing Framework for IPv6 Configuration Protocols

“…In [10], the major algorithms are explained. The authors of [7] propose to use graph kernels and gaussian processes for relational reinforcement learning.…”

Relevance Grounding for Planning in Relational Domains

“…The idea is to describe important world features in terms of abstract logical formulas enabling generalization over objects and situations. Examples of model-free approaches employ relational regression trees [8] or instance-based regression using distance metrices between relational states such as graph kernels [7] to learn Q-functions. Modelfree approaches have the disadvantage to be inflexible as they enable planning only for the specific problem type used in the training examples.…”

Learning to Repair Plans and Schedules Using a Relational (Deictic) Representation

“…Initialize the Q-function hypothesis 0 Q e ← 0 repeat Examples ← ∅ Generate a starting schedule state s 0 i ← 0 repeat choose a repair operator a i at s i using a policy (e.g., ε-greedy) based on the current hypothesis ê Q implement operator a i , observe r i and the resulting schedule s i+1 i ← i +1 until schedule state s i is a goal state for j =i -1 to 0 do generate example Several incremental relational regression techniques have been developed that meet the above requirements for RRL implementation: an incremental relational tree learner TG , an instance based learner , a kernel-based method (Gärtner et al, 2003;Driessens et al, 2006) and a combination of a decision tree learner with an instance-based learner (Driessens and Džeroski, 2004). Of these algorithms, the TG is the most popular one, mainly because it is relatively easy to specify background knowledge in the form of a language bias.…”

“…Of these algorithms, the TG is the most popular one, mainly because it is relatively easy to specify background knowledge in the form of a language bias. In the other methods, it is necessary to specify a distance function between modeled objects (Gärtner, 2008) or a kernel function is needed between (state, action)-pairs (Driessens et al, 2006). …”