Novel Metrics to Evaluate Probabilistic Remaining Useful Life Prognostics with Applications to Turbofan Engines

Abstract

Well-established metrics such as the Root Mean Square Error or the Mean Absolute Error are not suitable to evaluate estimated distributions of the Remaining Useful Life (i.e., probabilistic prognostics). We therefore propose novel metrics to evaluate the quality of probabilistic Remaining Useful Life prognostics. We estimate the distribution of the Remaining Useful Life of turbofan engines using a Convolutional Neural Network with Monte Carlo dropout. The accuracy and sharpness of the obtained probabilistic prognostics are evaluated using the Continuous Ranked Probability Score (CRPS) and weighted CRPS. The reliability of the obtained probabilistic prognostics is evaluated using the α-Coverage and the Reliability Score. The results show that the estimated distributions of the Remaining Useful Life of turbofan engines are accurate, reliable and sharp when using a Convolutional Neural Network with Monte Carlo dropout. In general, the proposed metrics are suitable to evaluate the accuracy, sharpness and reliability of probabilistic Remaining Useful Life prognostics.

Dynamic predictive maintenance for multiple components using data-driven probabilistic RUL prognostics: The case of turbofan engines

“…The probability that engine 9 would fail before day 815 was estimated at 7.7%. The costs of using a generic slot (10) thus exceeded the expected failure costs (𝑐 f ⋅0.077 = 50⋅0.077 = 7.7). The replacement was therefore scheduled at day 815.…”

“…The RMSE only evaluates the mean RUL prediction. We therefore use the 𝛼-Coverage and the reliability diagram to evaluate the reliability of the PDF of the RUL, i.e., how well the predicted probabilities match with the observed outcomes [10,38]. The 𝛼-Coverage is defined as [10]:…”

“…The closer the 𝛼-Coverage is to 𝛼, the more reliable the RUL predictions are. Related to this metric is the 𝛼-Mean width, which is the mean width in flight cycles of the confidence intervals at 𝛼 [10,38]:…”

“…Fig. 5 shows the reliability diagram of the four subsets of C-MAPSS [10]. Here, 𝐶(𝛼) 𝑖 = {𝛼-Coverage, 𝛼 ∈ {0.00, 0.01, 0.02, … , 1.00}} is the reliability curve of subset 𝑖 ∈ {FD001, FD002, FD003, FD004}.…”