Opportunistic maintenance scheduling with deep reinforcement learning

Valet, Alexander; Altenmüller, Thomas; Waschneck, Bernd; May, Marvin Carl; Kuhnle, Andreas; Lanza, Gisela

doi:10.1016/j.jmsy.2022.07.016

Cited by 38 publications

(4 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…RL has several application areas in manufacturing, such as dynamic scheduling of tasks in cloud manufacturing [52], maintenance scheduling [53], path planning of automated guided vehicles [54], or human-robot interaction [55]. Furthermore, RL has been successfully used for multi objective optimization [56].…”

Section: Reinforcement Learningmentioning

confidence: 99%

Multi objective factory layout planning using simulation-based reinforcement learning

Klar¹,

Schworm²,

Wu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Factory layout planning aims at finding an optimized layout configuration under consideration of varying influences such as the material flow characteristics. Manual layout planning can be characterized as a complex decision-making process due to a large number of possible placement options. Automated planning approaches aim at reducing the manual planning effort by generating optimized layout variants in the early stages of layout planning. Recent developments have introduced Reinforcement Learning (RL) based planning approaches that allow to optimize a layout under consideration of a single optimization criterion. However, within layout planning, multiple partially conflicting planning objectives have to be considered. Such multiple objectives are not considered by existing RL-based approaches. This paper addresses this research gap by presenting a novel RL-based layout planning approach that allows consideration of multiple objectives for optimization. Furthermore, existing RL-based planning approaches only consider analytically formulated objectives such as the transportation distance. Consequently, dynamic influences in the material flow are neglected which can result in higher operational costs of the future factory. To address this issue, a discrete event simulation module is developed that allows simulating manufacturing and material flow processes simultaneously for any layout configuration generated by the RL approach. Consequently, the presented approach considers material flow simulation results for multi-objective optimization. In order to investigate the capabilities of RL-based factory layout planning, different RL architectures are compared based on a simplified application scenario. In terms of optimization objectives, the throughput time, media supply, and clarity of the material flow are considered. The best performing architecture is then applied to an industrial planning scenario with 43 functional units to illustrate the approach. Furthermore, the performance of the RL approach is compared to the manually planned layout and to the results generated by a combined version of the genetic algorithm and tabu search. The results indicate that the RL approach is capable of improving the manually planned layout significantly. Furthermore, it reaches comparable results for the throughput time and better results for the clarity of the material flow compared to the combined version of a genetic algorithm and tabu search.

show abstract

Section: Reinforcement Learningmentioning

confidence: 99%

Multi objective factory layout planning using simulation-based reinforcement learning

Klar¹,

Schworm²,

Wu³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…RL has several application areas in manufacturing, such as dynamic scheduling of tasks in cloud manufacturing [56], maintenance scheduling [57], path planning of automated guided vehicles [58], or human-robot interaction [59]. Furthermore, RL has been successfully used for multi-objective optimization [60].…”

Section: Reinforcement Learningmentioning

confidence: 99%

Transferable multi-objective factory layout planning using simulation-based deep reinforcement learning

Klar,

Schworm,

et al. 2023

Preprint

View full text Add to dashboard Cite

Factory layout planning aims at finding an optimized layout configuration under consideration of varying influences such as the material flow characteristics. Manual layout planning can be characterized as a complex decision-making process due to a large number of possible placement options. Automated planning approaches aim at reducing the manual planning effort by generating optimized layout variants in the early stages of layout planning. Recent developments have introduced deep Reinforcement Learning (RL) based planning approaches that allow to optimize a layout under consideration of a single optimization criterion. However, within layout planning, multiple partially conflicting planning objectives have to be considered. Such multiple objectives are not considered by existing RL-based approaches. This paper addresses this research gap by presenting a novel deep RL-based layout planning approach that allows consideration of multiple objectives for optimization. Furthermore, existing RL-based planning approaches only consider analytically formulated objectives such as the transportation distance. Consequently, dynamic influences in the material flow are neglected which can result in higher operational costs of the future factory. To address this issue, a discrete event simulation module is developed that allows simulating manufacturing and material flow processes simultaneously for any layout configuration generated by the RL approach. Consequently, the presented approach considers material flow simulation results for multi-objective optimization. To investigate the capabilities of RL-based factory layout planning, different RL architectures are compared based on a simplified application scenario. Throughput time, media supply, and material flow clarity are considered as optimization objectives. The best performing architecture is then applied to an exemplary application scenario and compared with the results obtained by a combined version of the genetic algorithm and tabu search, the non-dominated sorting genetic algorithm, and the optimal solution. Finally, an industrial planning scenario with 43 functional units is considered. The results show that the performance of RL compared to meta-heuristics depends on the considered computation time. It is found that meta-heuristics lead to superior results in the early computation phase. However, with time, RL achieves comparable results for throughput time and better results for material flow clarity. Finally, the potential of applying transfer learning is investigated for three different application scenarios. It is observed that RL can learn generalized patterns for factory layout planning, which allows to significantly reduce the required training time and can lead to improved solution quality.

show abstract

“…It follows that this occurs in at most |J | iterations of the schedule repair as the re-entrant machine can only have |J | higher pass operations to be re-ordered. At this point, Equation (8) becomes…”

Section: ) Safe Maintenance Policiesmentioning

confidence: 99%

“…Sometimes, this dependence can be ignored and solutions can focus on preventive or policy based maintenance [5]- [7]. Recent work in this direction has used reinforcement learning to come up with these policies [8]. In other cases, the maintenance and production planning problem can be so integrated that the effect of use patterns on maintenance cannot be ignored.…”

Section: Introductionmentioning

confidence: 99%

Sequence- and Time-Dependent Maintenance Scheduling in Twice Re-Entrant Flow Shops

Eigbe,

De Schutter,

Nasri

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Industrial and academic interest converge on scheduling flow shops with sequence-and timedependent maintenance. We posit that anticipatory, integrated scheduling of operational and maintenance tasks leads to superior performance to purely 'wait-then-fix' handling of the maintenance tasks. Motivated by an industrial problem with (sequence dependent) setup times, maximum separation constraints, and a combination of sequence-and time-dependent maintenance tasks, this paper introduces an integer programming solution, a constraint programming solution and a heuristic solution based on list scheduling. The motivating use case provides a unique combination of concerns that is to the best of our knowledge, not yet studied in the literature. We build on existing work where we can by extending models for sequence-dependent maintenance scheduling to accommodate sequence-and time-dependent maintenance scheduling and also propose other new models. We show the relative performances of our methods through empirical evaluations and also show significant improvements -up to 25% reduction in makespan -when compared to a reactive scheduling approach that does not consider maintenance in its planning. Based on our evaluations on exact methods, constraint programming models scale better than mixed integer programming models for this problem.INDEX TERMS Flexible manufacturing systems, maintenance scheduling, makespan minimisation, re-entrant flow shops.

show abstract

Opportunistic maintenance scheduling with deep reinforcement learning

Cited by 38 publications

References 46 publications

Multi objective factory layout planning using simulation-based reinforcement learning

Multi objective factory layout planning using simulation-based reinforcement learning

Transferable multi-objective factory layout planning using simulation-based deep reinforcement learning

Sequence- and Time-Dependent Maintenance Scheduling in Twice Re-Entrant Flow Shops

Contact Info

Product

Resources

About