Safe Model-based Control from Signal Temporal Logic Specifications Using Recurrent Neural Networks

Abstract

We propose a policy search approach to learn controllers from specifications given as Signal Temporal Logic (STL) formulae. The system model is unknown, and it is learned together with the control policy. The model is implemented as a feedforward neural network (FNN). To capture the history dependency of the STL specification, we use a recurrent neural network (RNN) to implement the control policy. In contrast to prevalent model-free methods, the learning approach proposed here takes advantage of the learned model and is more efficient. We use control barrier functions (CBFs) with the learned model to improve the safety of the system. We validate our algorithm via simulations. The results show that our approach can satisfy the given specification within very few system runs, and therefore it has the potential to be used for on-line control.

STL2vec: Signal Temporal Logic Embeddings for Control Synthesis With Recurrent Neural Networks

“…In Problem 1, we look for a set of the parameters of the control policy W that maximizes the sum of the expectation of the robustness score with respect to the distribution of initial states over all M candidate STL specifications. Since the control policy depends on the past and present states, or it is history dependent, in this paper we use a recurrent neural network (RNN) to learn the control policy π (see, [14], [15]). RNN is a type of a neural network that has a feedback architecture.…”

“…Remark 3: When we work with a long temporal sequence of states, numerical instability can be problematic with the above scheme since when computing the gradient of the cost function, the derivative computed at the previous time step should be multiplied with a Jacobian matrix of the system model. To address this problem, [11], [15] employed the weights update scheme with the Lagrange Multiplier method. Although we do not consider these techniques in this paper since such a problem was not observed in the example discussed in Section VI, it may be desirable to use such techniques for applications with a long time horizon.…”

“…Aiming to further improve the scalability and meet realtime requirements, neural network (NN) based feedback controller was employed for STL tasks [11]- [15]. Although the learning procedure itself may be time-consuming, once this computation can be done offline, applying the resulting controller online becomes much faster than directly solving Wataru Hashimoto, Kazumune Hashimoto and Shigemasa Takai are with the Graduate School of Engineering, Osaka University, Suita, Japan (e-mail: hashimoto@is.eei.eng.osaka-u.ac.jp, hashimoto@eei.eng.osakau.ac.jp, takai@eei.eng.osaka-u.ac.jp) the optimization problem at each time step.…”

“…Reinforcement learning algorithms such as deep Q-learning (DQN) and learning from demonstrations (Lfd) were used for learning the NN controller in [12] and [13] respectively. The works of [14], [15] used a recurrent neural network (RNN) [16] instead of a feed-forward NN, which is more suitable for control with STL specifications since the satisfaction of an STL specification depends on a state trajectory (not just the current state). In [14], [15], RNN parameters were trained using imitation learning and model-based reinforcement, respectively.…”

DeepSTL -- From English Requirements to Signal Temporal Logic

“…We analyzed over 130 STL specifications and their associated English-language formulation, from scientific papers and industrial documents. The investigated literature covers multiple application domains: specification patterns [42], automatic driving [43][44][45], robotics [46][47][48][49][50], time-series analysis [51] and electronics [2,52]. Although this literature contains data that is not statistically exhaustive, it still provides valuable information to guide the design of the data generator and address the research question RQ1.…”