Practical Methods of Optimization

a b s t r a c tAutonomous learning is one of the hallmarks of human and animal behavior, and understanding the principles of learning will be crucial in order to achieve true autonomy in advanced machines like humanoid robots. In this paper, we examine learning of complex motor skills with human-like limbs. While supervised learning can offer useful tools for bootstrapping behavior, e.g., by learning from demonstration, it is only reinforcement learning that offers a general approach to the final trial-and-error improvement that is needed by each individual acquiring a skill. Neither neurobiological nor machine learning studies have, so far, offered compelling results on how reinforcement learning can be scaled to the high-dimensional continuous state and action spaces of humans or humanoids. Here, we combine two recent research developments on learning motor control in order to achieve this scaling. First, we interpret the idea of modular motor control by means of motor primitives as a suitable way to generate parameterized control policies for reinforcement learning. Second, we combine motor primitives with the theory of stochastic policy gradient learning, which currently seems to be the only feasible framework for reinforcement learning for humanoids. We evaluate different policy gradient methods with a focus on their applicability to parameterized motor primitives. We compare these algorithms in the context of motor primitive learning, and show that our most modern algorithm, the Episodic Natural Actor-Critic outperforms previous algorithms by at least an order of magnitude. We demonstrate the efficiency of this reinforcement learning method in the application of learning to hit a baseball with an anthropomorphic robot arm.

show abstract

“…then have a restricted step-size gradient descent problem (Fletcher & Fletcher, 2000). Thus, we have an optimization problem…”

Section: Motivationmentioning

confidence: 99%

Reinforcement learning of motor skills with policy gradients

2008

View full text Add to dashboard Cite

show abstract

“…This iterative procedure may take a relatively small computational time since the summation term in (7) is conduced only for a set of support vectors Ω due to Karush-Kuhn-Tucker dual complementarity condition [19]. Therefore, with a low cost, ATLAS adaptively determines time-step sizes and methods in accordance with the specified level of the required local convergence.…”

Section: Adaptive Time-stepping With Support Vector Machinesmentioning

confidence: 99%

An Adaptive Time-Stepping Scheme with Local Convergence Verification Using Support Vector Machines

Kawaguchi¹,

Ishikawa²,

Maruyama³

2014

IJET

View full text Add to dashboard Cite

Abstract-An adaptive time-stepping scheme in accordance with the local convergence of computation often involves computationally expensive procedures. As a result, many computer simulators have avoided utilizing such an adaptive scheme, while its advantages are well recognized; the scheme not only efficiently allocates computational resources, but also makes the results of the computation more reliable. In this paper, we propose a fast adaptive time-stepping scheme, ATLAS (Adaptive Time-step Learning and Adjusting Scheme), which approximates such an expensive yet beneficial scheme by using support vector machines (SVMs). We demonstrate that ATLAS performs quite favorably when compared with computations without it. ATLAS can incorporate existing solvers and other fast but unreliable adaptive schemes to meet the different criteria required in various applications.Index Terms-Adaptive time step control, machine learning, ordinary differential equations, severe accident analysis.

show abstract

“…Similar computational needs and problems with large numbers of parameters apply for the maximum likelihood method which is used for single channel measurements as in the software package QuB (Milescu et al 2005). Sequential quadratic programming methods also can be used to find a constrained minimum with superlinear convergence (Gill 1981, Fletcher 1987 for model parameters using Matlab™ (Bueno-Orovio et al 2008). Global methods like genetic algorithms, which have been used to fit conductance values (Syed et al 2005), simulated annealing (e.g.…”

Section: Parameterisation Data Availability and Local Minimamentioning

confidence: 99%

Cardiac cell modelling: Observations from the heart of the cardiac physiome project

Fink

Niederer

Cherry

et al. 2011

Progress in Biophysics and Molecular Biology

149

136

View full text Add to dashboard Cite

In this manuscript we review the state of cardiac cell modelling in the context of international initiatives such as the IUPS Physiome and Virtual Physiological Human Projects, which aim to integrate computational models across scales and physics. In particular we focus on the relationship between experimental data and model parameterisation across a range of model types and cellular physiological systems. Finally, in the context of parameter identification and model reuse within the Cardiac Physiome, we suggest some future priority areas for this field.

show abstract

Practical Methods of Optimization

Cited by 1,935 publications

References 0 publications

Reinforcement learning of motor skills with policy gradients

Reinforcement learning of motor skills with policy gradients

An Adaptive Time-Stepping Scheme with Local Convergence Verification Using Support Vector Machines

Cardiac cell modelling: Observations from the heart of the cardiac physiome project

Contact Info

Product

Resources

About