Analyzing Real Options and Flexibility in Engineering Systems Design Using Decision Rules and Deep Reinforcement Learning

Abstract

Engineering systems provide essential services to society e.g., power generation, transportation. Their performance, however, is directly affected by their ability to cope with uncertainty, especially given the realities of climate change and pandemics. Standard design methods often fail to recognize uncertainty in early conceptual activities, leading to rigid systems that are vulnerable to change. Real Options and Flexibility in Design are important paradigms to improve a system's ability to adapt and respond to unforeseen conditions. Existing approaches to analyze flexibility, however, do not leverage sufficiently recent developments in machine learning enabling deeper exploration of the computational design space. There is untapped potential for new solutions that are not readily accessible using existing methods. Here, a novel approach to analyze flexibility is proposed based on Deep Reinforcement Learning (DRL). It explores available datasets systematically and considers a wider range of adaptability strategies. The methodology is evaluated on an example waste-to-energy system. Low and high flexibility DRL models are compared against stochastically optimal inflexible and flexible solutions using decision rules. The results show highly dynamic solutions, with action space parametrized via artificial neural network. They show improved expected economic value up to 69% compared to previous solutions. Combining information from action space probability distributions along expert insights and risk tolerance helps make better decisions in real-world design and system operations. Out of sample testing shows that the policies are generalizable, but subject to tradeoffs between flexibility and inherent limitations of the learning process.

Incorporating changeability for value-robust product-service systems: an integrative review

“…The decision matrices can scale exponentially as the system's complexity increases, making the evaluation of policies intractable. Deep reinforcement learning is an approximation technique based on the MDP that can mitigate these dimensionality issues (Caputo & Cardin 2021). In reinforcement learning, an agent interacts with its so-called environments, takes some actions, transitions to the next state and gains rewards for choosing that action.…”

“…The term "deep" implies using neural networks for this approximation. Caputo & Cardin (2021) used deep reinforcement learning on a waste-to-energy system example and showed better valuation performance than decision rules. This approach opens the possibility of investigating the potential of many different deep reinforcement learning algorithms for changeability valuation.…”

“…Thus, a path-dependent arrangement of these external factors is vital in developing a value-robust system. Path dependency mainly concerns the irreversibility of decisions made at the stage gates of external changes (Caputo & Cardin 2021). For example, the decision to expand a system to meet demand higher than expectations may not be easily reverted if the market falls in the future.…”

“…These methods typically use evolutionary optimization algorithms for ranking promising options to be embedded in the system and finding a set of conditions denoting when the change option shall be exercised. These methods may also be referred to as robust optimization (Caputo & Cardin 2021) since the optimization problem involves finding a pair comprising the option and the decisions that maximize value. In the evolutionary real options framework, Zhang & Babovic (2011) used an evolutionary algorithm to assess change options and the exercising decision by fine-tuning the exercising condition thresholds.…”

Distributional Reinforcement Learning for Scheduling of Chemical Production Processes

“…Although RL has been demonstrated for sequential decision making in a number of case studies [21,22,23,24], its application to physical production systems has been relatively limited. For example, [25] applied deep Q networks to optimize a flexible jobshop (i.e.…”