Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints

Abstract

Determination of inspection and maintenance policies for minimizing long-term risks and costs in deteriorating engineering environments constitutes a complex optimization problem. Major computational challenges include the (i) curse of dimensionality, due to exponential scaling of state/action set cardinalities with the number of components; (ii) curse of history, related to exponentially growing decision-trees with the number of decisionsteps; (iii) presence of state uncertainties, induced by inherent environment stochasticity and variability of inspection/monitoring measurements; (iv) presence of constraints, pertaining to stochastic long-term limitations, due to resource scarcity and other infeasible/undesirable system responses. In this work, these challenges are addressed within a joint framework of constrained Partially Observable Markov Decision Processes (POMDP) and multi-agent Deep Reinforcement Learning (DRL). POMDPs optimally tackle (ii)-(iii), combining stochastic dynamic programming with Bayesian inference principles. Multi-agent DRL addresses (i), through deep function parametrizations and decentralized control assumptions. Challenge (iv) is herein handled through proper state augmentation and Lagrangian relaxation, with emphasis on life-cycle risk-based constraints and budget limitations. The underlying algorithmic steps are provided, and the proposed framework is found to outperform wellestablished policy baselines and facilitate adept prescription of inspection and intervention actions, in cases where decisions must be made in the most resource-and risk-aware manner.

Deep reinforcement learning for predictive aircraft maintenance using probabilistic Remaining-Useful-Life prognostics

“…Recently, studies have proposed deep reinforcement learning (DRL) for adaptive maintenance planning [32], where no fixed thresholds are needed to schedule maintenance. In [36], DRL is used to schedule the replacement of a component whose degradation is represented by 4 discrete states. In [37], the maintenance of multi-component systems is optimised using DRL, where they assume that the degradation follows a compound Poisson process and a Gamma process.…”

Inspection and maintenance planning for offshore wind structural components: integrating fatigue failure criteria with Bayesian networks and Markov decision processes

“…To overcome this curse of dimensionality, further research efforts are suggested towards the integration of the FAD criterion with POMDP-based deep reinforcement learning (DRL) approaches. The capability of POMDP-based DRL approaches to efficiently provide optimal I&M strategies for large state space problems has been demonstrated in Andriotis and Papakonstantinou (2021).…”

A Novel Deep Reinforcement Learning Model for Resilient Road Network Recovery from Multiple Hazards

“…The deep reinforcement learning algorithm is seen as the most promising method. It has been used for multiple engineering applications such as inspection and maintenance planning [20,21], satellite communications [22], and production systems [23]. Although many studies have demonstrated its ability in tackling the optimization problems with high dimensional decision space and state space [24][25][26][27], few studies have used it for the emergency management, such as the decisions in the recovery process [28].…”