Imitation learning with non-parametric regression

Abstract

Abstract-Humans are very fast learners. Yet, we rarely learn a task completely from scratch. Instead, we usually start with a rough approximation of the desired behavior and take the learning from there. In this paper, we use imitation to quickly generate a rough solution to a robotic task from demonstrations, supplied as a collection of state-space trajectories. Appropriate control actions needed to steer the system along the trajectories are then automatically learned in the form of a (nonlinear) statefeedback control law. The learning scheme has two components: a dynamic reference model and an adaptive inverse process model, both based on a data-driven, non-parametric method called local linear regression. The reference model infers the desired behavior from the demonstration trajectories, while the inverse process model provides the control actions to achieve this behavior and is improved online using learning. Experimental results with a pendulum swing-up problem and a robotic arm demonstrate the practical usefulness of this approach. The resulting learned dynamics are not limited to single trajectories, but capture instead the overall dynamics of the motion, making the proposed approach a promising step towards versatile learning machines such as future household robots, or robots for autonomous missions.

Stiffness and damping scheduling for legged locomotion

“…where σ s is a factor determining the length of the running average that was set to a value of 0.95, as in [12]. A low value for η s means that the sample and its closest neighbors live in the neighborhood of the same affine hyperplane.…”

“…The strategy implemented in [12] to reduce noise is to replace the nearest neighbors y s with the estimated values using the local modelĥ(x s ). The local model is a linear least squares solution of the nearest neighbors, which is the best average of the first order relation present in the neighbors.…”

“…The memory management algorithm implemented in [12] is based on discarding new samples (or replacing old similar samples with the new observation) that fit best with the current affine model generated by the LLR. In practice this means that the nonlinear parts are populated with many samples while the linear parts have very little.…”

Human Demonstrations for Fast and Safe Exploration in Reinforcement Learning

“…Another method to improve the speed of learning is the use of human demonstrations to explore the state space of a system. Several attempts have been made to do so, 7,8 however, these attempts did use a model of the dynamics of the system at hand. A method that accelerates the reinforcement learning in a model-free and online fashion through the use of human demonstrations, has not yet been found.…”

“…Successful attempts have been made to include knowledge on the dynamics of the system to speed up the learning. 6,7 These attempts used different kinds of knowledge on the system dynamics to do so. Another method to improve the speed of learning is the use of human demonstrations to explore the state space of a system.…”