Temporal-difference Q-learning in active fault diagnosis

Škach, Jan; Punčochář, Ivo; Lewis, Frank L.

doi:10.1109/systol.2016.7739765

Cited by 4 publications

(6 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are two types of the tasks involved in RL. One is 'prediction,' and the other is 'control' [131]. The prediction task predicts the total expected reward from any given state, assuming function policy π(a|s) is given.…”

Section: Figure 210 the Rl Modelmentioning

confidence: 99%

Nonlinear Filtering and Reinforcement Learning-based Smart Autonomous Multi-agent Systems

Borah

2024

Preprint

View full text Add to dashboard Cite

<p>There is growing importance in complex engineering systems to operate autonomously, especially given potential malfunctions that may occur in the system, such as sensors, actuators, components, communication networks, and controllers. Fault detection, isolation, and reconstruction (FDIR) are crucial for autonomous systems. There is significant demand to evolve efficient intelligent systems to detect faults, isolate fault locations and autonomously reconstruct any component of a complex dynamical system. Hence, it is essential to detect, isolate, and reconstruct the faults efficiently and on time when the systems are in operation. This dissertation presents a novel methodology for developing smart autonomous multiagent systems (SAMAS). The SAMAS comprises several agents; some are homogeneous while others are heterogeneous. The proposed method involves developing a multi-agent systems (MAS) model and designing a decentralized smart control system. The MAS model contains homogeneous and heterogeneous agents, communicates among agents through an undirected connected graph, external disturbances, goals, and constraints. The sensor, actuator, communication, and controller faults are also modeled in the MAS model. A decentralized smart control system is designed to create a SAMAS model in the presence of uncertainty in each agents’ dynamics and faults located in the sensors, actuators, communication networks, and controllers. The proposed control method is based on non-linear filtering techniques, deep reinforcement learning, and robust control techniques. A Chebyshev neural network (CNN) is incorporated to learn the uncertain nonlinear functions in the agent dynamics of MAS. Additionally, robust control term using the hyperbolic tangent function is applied to counteract the neural network approximation errors. Meanwhile, a novel algorithm has been proposed which is employed to estimate the uncertain states of the agent dynamics and to train the internal parameters of the neural network given a set of prior measurements. Moreover, an adaptive threshold method has been proposed to detect any kinds of faults present in the system followed by a likelihood-based isolation method to locate each faulty agent. The novel algorithm is known as a reinforced unscented Kalman filter (RUKF). The primary purpose of the RUKF is to detect and isolate the faults, and to adapt the process and measurement noise covariance matrices to reconstruct the faults. To assess the performance of the proposed methodology, we developed a SAMAS consisting of six heterogeneous uncertain agent dynamics. The SAMAS model and the proposed control methodology are numerically simulated using MATLAB. A Monte-Carlo (MC) simulation was carried out to assess the performance of the proposed control methodology in the presence of uncertain agent dynamics and with the sensor, actuator, communication, and controller faults. The fault isolation results are summarized in confusion matrices for each faulty case. The stability of the RUKF, which ran in conjunction with a robust control method, has been proven using the Lyapunov stability approach. Extensive simulations were conducted to evaluate the performance of the proposed method. In this study, the proposed method showed superior performance to the standard unscented Kalman filter (UKF) and adaptive UKF (AUKF). The proposed fault isolation scheme isolated the faulty agents with over 96.68% success rate at the system level. Hence, the results of the numerical simulations show the feasibility of the proposed approach. The proposed RUKF is also computationally less expensive than the standard UKF and AUKF. The proposed approach can be considered a promising tool to evaluate fault detection, isolation, and reconstruction in complex engineering systems. Furthermore, the proposed approach can be extended to other deep space complex systems. The proposed approach can also be used for MAS problems for the financial sector, such as stock market prediction, economic time series, and multi-arm bandit problems.</p>

show abstract

Section: Figure 210 the Rl Modelmentioning

confidence: 99%

Nonlinear Filtering and Reinforcement Learning-based Smart Autonomous Multi-agent Systems

Borah

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…when s ̸ = 0, the identifier (10) represents the fault identifier, and when s = 0, the identifier (10) represents the normal identifier. Compare (10) with (9), and define…”

Section: Dlt-based System Identificationmentioning

confidence: 99%

“…The critic NN is applied to approximate the strategic utility function, that is, to estimate the long-term system performance [18] [19]. Interesting results about the ADP/actor-critic methods for the dynamical system have been reported in the recent literatures, and have achieved excellent performance in FD [10]- [13]. For example, in [10], inspired from the ADP techniques, an active fault detector is designed for a class of discrete-time system, where a reward function is set to impose the penalty of the wrong fault detection decisions.…”

Section: Introductionmentioning

confidence: 99%

“…Interesting results about the ADP/actor-critic methods for the dynamical system have been reported in the recent literatures, and have achieved excellent performance in FD [10]- [13]. For example, in [10], inspired from the ADP techniques, an active fault detector is designed for a class of discrete-time system, where a reward function is set to impose the penalty of the wrong fault detection decisions. In [11], an ADP-based active fault detector is employed for stochastic linear Markovian switching systems over an infinite time horizon to improve the discrimination between models.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Fault Diagnosis for A Class of Nonlinear Systems Based on Reinforcement Learning and Deterministic Learning

Zhu¹,

Wu²,

Chen³

2022

Preprint

View full text Add to dashboard Cite

In this paper a novel fault diagnosis (FD) approach combing the rein-forcement learning (RL) and the deterministic learning theory (DLT)is proposed for a class of discrete-time nonlinear system with unknowndynamics. First, a bank of DLT-based dynamical neural network (NN)identifiers are utilized to achieve locally-accurate approximations ofthe unknown system dynamics along the normal and fault trajecto-ries. Based on this, a novel feature learning method combing the RLwith DLT is proposed to further adapt the NN weights to extract dis-criminative features. The extracted features are represented by constant NNs. Finally, constant NN-based dynamical estimators are constructedto achieve rapid FD. The novelties of the proposed methods are: 1)according to the DLT, the exponential convergence of the NN weightscan be rigorously analysed based on the Lyapunov stability theory;2) a new class of strategic utility function is designed based on theconcept of the synchronization error in DLT-based dynamical pat-tern recognition approach, which is different from other RL-based FDtechniques that penalise the future wrong FD decisions. Simulationresults shows the practical significance of the proposed FD method.

show abstract

“…Skach et al [12], have utilized the RL method for fault detection and isolation in a system with the uncertainty. Yih et al [13] have combined the RL and Fuzzy Logic to detect system faults.…”

Section: Introductionmentioning

confidence: 99%

Using Sliding Mode Controller and Eligibility Traces for Controlling the Blood Glucose in Diabetic Patients at the Presence of Fault

Noori

Sadrnia

Sistani

2020

IJE

View full text Add to dashboard Cite

Some people suffering from diabetes use insulin injection pumps to control the blood glucose level. Sometimes, the fault may occur in the sensor or actuator of these pumps. The main objective of this paper is controlling the blood glucose level at the desired level and fault-tolerant control of these injection pumps. To this end, the eligibility traces algorithm is combined with the sliding mode control. The eligibility traces algorithm is one of the newest solving methods of the Reinforcement Learning approach. The major disadvantage of the sliding mode control method is the chattering phenomenon. In this paper, the novel idea is the combination of these methods to remove the chattering phenomena in simulation results. To demonstrate the superiority of the proposed method, it is compared with another combinatory method that is the sliding mode control and Artificial Neural Networks. Simulation results reveal that the combination of the eligibility traces algorithm and the sliding mode control can control the blood glucose level and insulin with a higher speed and bring them to the desired level, even in the case the sensor and actuator faults are present in the system. When the proposed hybrid method is used, the injected dosage of the drug is lower, which will result in reduced side effects. Finally, the noise, as well as the uncertainty in system parameters and initial conditions are applied to the system to investigate the performance of the proposed controller, under the faulty condition.

show abstract

Temporal-difference Q-learning in active fault diagnosis

Cited by 4 publications

References 20 publications

Nonlinear Filtering and Reinforcement Learning-based Smart Autonomous Multi-agent Systems

Nonlinear Filtering and Reinforcement Learning-based Smart Autonomous Multi-agent Systems

Fault Diagnosis for A Class of Nonlinear Systems Based on Reinforcement Learning and Deterministic Learning

Using Sliding Mode Controller and Eligibility Traces for Controlling the Blood Glucose in Diabetic Patients at the Presence of Fault

Contact Info

Product

Resources

About