DOB-Net: Actively Rejecting Unknown Excessive Time-Varying Disturbances

Abstract

This paper presents an observer-integrated Reinforcement Learning (RL) approach, called Disturbance OBserver Network (DOB-Net), for robots operating in environments where disturbances are unknown and time-varying, and may frequently exceed robot control capabilities. The DOB-Net integrates a disturbance dynamics observer network and a controller network. Originated from classical DOB mechanisms, the observer is built and enhanced via Recurrent Neural Networks (RNNs), encoding estimation of past values and prediction of future values of unknown disturbances in RNN hidden state. Such encoding allows the controller generate optimal control signals to actively reject disturbances, under the constraints of robot control capabilities. The observer and the controller are jointly learned within policy optimization by advantage actor critic. Numerical simulations on position regulation tasks have demonstrated that the proposed DOB-Net significantly outperforms a canonical feedback controller and classical RL algorithms.

Modular Transfer Learning with Transition Mismatch Compensation for Excessive Disturbance Rejection

“…However, this is not the case when the disturbances are considered as the unobservable parts of the state space, since it it difficult to formulate a transition function to predict next disturbances from current state (including disturbances) and action only. Both history window approach [31] and recurrent policy [32] attempt to resolve this issue through characterizing the disturbed system transition as a multi-step MDP, and assuming the unobservable disturbance waveforms are encoded in robot motion history. The difference lies in the way to use the history data, the history window approach directly takes most recent state-action pairs as additional input to the policy, while the recurrent policy employs RNN to explore past experience in order to learn an optimal embedding of history data.…”

“…Previous work [32] has demonstrated that RNN can directly learn to control a dynamical system with unobservable disturbances in an end-to-end mode, where the past motion history is mapped to the control action. While, inspired by [33], this work applies modular learning procedures, that explicitly decouple the process into disturbance identification and motion control.…”

“…After several iterations (4 in our case), the disturbance rejection capability of the robot reaches a relatively high level, even approaching the performance of GCP when given the true disturbance parameters μ. In addition, we also evaluate an endto-end learning framework for excessive disturbance rejection, called Disturbance Observer Network (DOB-Net) [32], as a comparison with the modular architecture of GCP-ODI. We found that DOB-Net also achieves an excellent stability under the same task settings, but not as good as the iteratively trained GCP-ODI (4th iteration).…”

Including Image-based Perception in Disturbance Observer for Warehouse Drones

“…In [16], ILC is used to generate a correction signal for DOB to enhance disturbance attenuation when the major component of the disturbance is repetitive. Besides, neural networks have also been introduced to enhance DOB's performance [17][18][19][20]. For example, in [17], a radial basis function NN is combined with DOB to deal with both unknown dynamics and external disturbances; in [20], the conventional DOB is enhanced via Recurrent Neural Networks for disturbance estimation and prediction.…”

“…Moreover, the performance of conventional DOB depends highly on an accurate plant inverse which usually is not available or is very sensitive to uncertainties, and this significantly limits DOB's performance. Recently, deep learning techniques have been developed and applied to highlevel decision making (e.g., [21][22][23]) and low-level trajectory planning and tracking (e.g., [20,[24][25][26]). Since the drone delivery scenarios considered in this paper is relatively structured, here we leverage the deep learning techniques in convolutional neural network (CNN) and long short-term memory (LSTM) network to include the image-based perception into the DOB framework, aiming to improve DOB's performance.…”