Characterization and Control of Hysteretic Dynamics Using Online Reinforcement Learning

Abstract

Hysteretic dynamical systems are challenging to control due to their hard nonlinearity and difficulty in modeling. One type of system with hysteretic dynamics that is gaining use in aerospace systems is the shape-memory alloy-based actuator. These actuators provide aircraft and spacecraft systems with the ability to achieve component-level or vehicle-level geometry or shape changes. Characterization of the material dynamics and properties of these actuators is usually accomplished with empirical testing of physical specimens, in which the hysteresis dynamics are often abstracted to very simplified models or ignored entirely. Machine learning techniques have the potential to learn hysteretic dynamics, but they routinely encounter difficulties that make them unsuitable. This paper proposes and develops a reinforcement learning-based approach that directly learns an input-output mapping characterization of hysteretic dynamics, which is then used as a control policy. A hyperbolic tangent-based model is used to develop a simulation of a shape-memory alloy, which is then validated experimentally using the Sarsa algorithm. The simulation model produces the temperature-versus-strain behavior and characterizes both the major and minor hysteresis loops. The learning results produce a near-optimal control policy for modulating a shape-memory alloy wire to a specified length. Results presented in the paper show that casting the shape-memory alloy control problem as a reinforcement learning problem shows promise for characterizing and controlling shape-memory alloy hysteresis behavior.

Asymmetric Airfoil Morphing via Deep Reinforcement Learning

“…Subsequently, we apply the proposed method to track optimized airfoil shapes in different flight conditions and show the morphing procedures. The values of parameters in our simulation are given in Table 1 [ 40 , 41 ].…”

“…The transitions to martensite and austenite have different start and end temperatures, which leads to the hysteresis properties of the strain with respect to the temperature. Instead of common methods such as Preisach model and Krasnosel’skii–Pokrovskii model [ 49 ], the SMA hysteresis is characterized using hyperbolic tangent functions for their efficiency in computation and accuracy in curve fitting [ 41 ]. The strain is replaced by a radius factor equivalently such that .…”

“…For heating and cooling starting with temperatures outside the transformation region, namely that the initial temperature is not between the end temperatures of the phase transformations, the hysteresis properties are modeled by the major hysteresis loops as where T denotes the temperature, and , , , , w , parameterize the shape of the curves. Values of such parameters are chosen to fit the experimental data of SMA [ 40 , 41 ]. The radius factor varies according to the curve when the temperature decreases and when the temperature increases.…”

“…This gives rise to increasing investigations of learning-based control for SMAs. Reinforcement learning methods including Q-learning and Sarsa have been applied to adjust the strain of SMAs via resistive heating, where the hysteretic dynamics are modeled as hyperbolic tangent curves [ 40 , 41 ]. However, there is no explicit consideration for the hysteresis properties in the policy-learning procedure of these studies.…”

Prediction model-based learning adaptive control for underwater grasping of a soft manipulator

“…comparing with dynamic programming and Monte Carlo (Sutton and Barto 1998;Sutton 1988;Kirkpatrick and Valasek 2009;Kirkpatrick et al 2013). The detailed design procedure is described as follows.…”