Designing reinforcement learning (RL) problems that can produce delicate and precise manipulation policies requires careful choice of the reward function, state, and action spaces. Much prior work on applying RL to manipulation tasks has defined the action space in terms of direct joint torques or reference positions for a joint-space proportional derivative (PD) controller. In practice, it is often possible to add additional structure by taking advantage of model-based controllers that support both accurate positioning and control of the dynamic response of the manipulator. In this paper, we evaluate how the choice of action space for dynamic manipulation tasks affects the sample complexity as well as the final quality of learned policies. We compare learning performance across three tasks (peg insertion, hammering, and pushing), four action spaces (torque, joint PD, inverse dynamics, and impedance control), and using two modern reinforcement learning algorithms (Proximal Policy Optimization and Soft Actor-Critic). Our results lend support to the hypothesis that learning references for a task-space impedance controller significantly reduces the number of samples needed to achieve good performance across all tasks and algorithms.

No abstract

No abstract

Hypertension, diabetes, and obesity are often associated with impaired microvascular function and structural adaptation. A.R. Pries and T.W. Secomb have developed a mathematical model of structural adaptation based on known physiological responses to shear stress, circumferential stress, and metabolic demand under healthy conditions. While this model captures key aspects of microvascular remodeling, it does not explicitly incorporate signaling pathways. As altered signaling pathways are a prominent feature of many disease states, in its current state this model cannot be used to predict the effects of diseases on vessel remodeling. Using the Pries and Secomb model as a framework, we have developed a model that incorporates relevant signaling pathways. In our model, diameter changes with nitric oxide, a vasodilator, and endothelin‐1, a vasoconstrictor, which are both functions of shear stress. Wall area changes with circumferential stress as well as the growth and death of vascular smooth muscle cells. Currently our model reflects the steady state trends in vessel geometry under normal conditions. Work is ongoing to further validate the model and examine vascular remodeling in disease states. Our long term goal is to improve the understanding of vascular adaptation in disease states and to create modeling tools which provide biologically testable hypotheses for experimentalists and clinicians.