Examining the Use of Temporal-Difference Incremental Delta-Bar-Delta for Real-World Predictive Knowledge Architectures

Abstract

Predictions and predictive knowledge have seen recent success in improving not only robot control but also other applications ranging from industrial process control to rehabilitation. A property that makes these predictive approaches well suited for robotics is that they can be learned online and incrementally through interaction with the environment. However, a remaining challenge for many prediction-learning approaches is an appropriate choice of prediction-learning parameters, especially parameters that control the magnitude of a learning machine's updates to its predictions (the learning rate or step size). To begin to address this challenge, we examine the use of online step-size adaptation using a sensor-rich robotic arm. Our method of choice, Temporal-Difference Incremental Delta-Bar-Delta (TIDBD), learns and adapts step sizes on a feature level; importantly, TIDBD allows step-size tuning and representation learning to occur at the same time. We show that TIDBD is a practical alternative for classic Temporal-Difference (TD) learning via an extensive parameter search. Both approaches perform comparably in terms of predicting future aspects of a robotic data stream. Furthermore, the use of a step-size adaptation method like TIDBD appears to allow a system to automatically detect and characterize common sensor failures in a robotic application. Together, these results promise to improve the ability of robotic devices to learn from interactions with their environments in a robust way, providing key capabilities for autonomous agents and robots.

Prediction, Knowledge, and Explainability: Examining the Use of General Value Functions in Machine Knowledge

“…Not only do these adaptive step-sizes improve learning, they can also provide useful information about the operation of an agent (Günther et al, 2019). Learned step-size values can also detect anomalies in the agent's observation stream, including those that indicate a hardware or sensor failure of the agent (Günther et al, 2020).…”

What’s a good prediction? Challenges in evaluating an agent’s knowledge

“…For example, Unexpected Demon Error (UDE), can be used to gauge how 'surprising' a given observation is to an agent (White, 2015). By examining the surprise, we can gauge how current experience relates to past experiencesfor example, detecting faults in a system (Günther et al, 2018).…”

Affordance as general value function: a computational model

“…Recent research has shown the benefit of predictions when used for prosthetic limbs (Gu¨nther et al, 2018(Gu¨nther et al, , 2020Pilarski et al, 2013;Sherstan & Pilarski, 2014), with Sherstan and Pilarski (2014) demonstrating that an abundant number of GVFs can be learned and maintained with reasonable computational resources, allowing these techniques to be used in mobile robots. Such GVF predictions have been demonstrated to be useful for prosthetic limbs in several ways.…”

“…The Modular Prosthetic Limb (MPL), a robot arm with many degrees of freedom and sensors used for research in prosthetic limbs (Günther et al, 2020). …”