A Systematic Methodology for Comparing Batch Process Monitoring Methods: Part I—Assessing Detection Strength

Abstract

A significant number of batch process monitoring methods have been proposed since the first groundbreaking approaches were published in the literature, two decades ago. The proper assessment of all the alternatives currently available requires a rigorous and robust assessment framework, in order to assist practitioners in their difficult task of selecting the most adequate approach for the particular situations they face and in the definition of all the optional aspects required, such as the type of preprocessing, infilling, alignment, etc. However, comparison methods currently adopted present several limitations and even some flaws that make the variety of studies available not easily generalizable or, in extreme situations, fundamentally wrong. Without a proper comparison tool decisions are made on a subjective basis and therefore are prone to be at least suboptimal. In this article we present a structured review of comparison methods and figures of merit adopted to assess batch process monitoring approaches, analyzing their strengths and limitations, as well as some common missuses. Furthermore, we propose a comparison and assessment methodology (CAM) and figures of merit that should be adopted in order to circumvent some of the limitations of the current procedures. We begin by addressing, in this article, the analysis of the methods' "detection strength". Detection strength regards the ability to correctly detect abnormal situations without resulting in excessive false alarms. We describe in detail how the comparison framework is implemented and illustrate its application in two case studies, encompassing a rich variety of testing scenarios (99 000 batch runs) and monitoring methods (2-way, 3-way, and dynamic methods, amounting to a total of 60 different combinations of methods and their variants). From this analysis, it stands out that the CAM methodology is suitable for comparing different monitoring methods, even when the number of methods and variants is very large. It provides simple, informative, and statisticalbased metrics to assess a given method's performance and is able to quantify the added value of alternative approaches. When applied to the two case studies considered, 2-way methods (with batch-wise unfolding) combined with missing data (MD) infilling and dynamic time warping (DTW) were found to provide adequate solutions. If a more parsimonious model is required, dynamic methods such as autoregressive principal components analysis (ARPCA) provide good alternatives and Tucker3 with current deviations (CD) infilling also presents an interesting performance for some fault scenarios. In general, no method can claim absolute supremacy in all faulty scenarios.

An extensive reference dataset for fault detection and identification in batch processes

“…Moita et al [58] proposed a framework hereto, which is currently further developed by Rato et al [69].…”

Challenges in the Development of Soft Sensors for Bioprocesses: A Critical Review

“…As with all, the choice of the data synchronization method is highly dependent on the process being monitored ( Rato et al, 2016 ; Rato et al, 2018 ). It should also be noted that, regardless of the method used for data synchronization, all subsequent levels of the monitoring algorithm (soft sensor prediction, fault detection, etc.)…”

“…Process variables that are used as termination criteria for the process or as trigger variables for an automation system are particularly suitable as indicator variables ( Ündey et al, 2003 ; García-Muñoz et al, 2011 ). Examples of process variables suitable as indicator variable are decrease of substrate concentration ( Ündey et al, 2002 ), cumulative feed volume ( Ündey et al, 2003 ), bioreactor volume, and biomass concentration ( Rato et al, 2016 ). Regardless whether a real process variable or a maturity index is used, the indicator variable should ideally progress strictly monotonically, continuously, and smoothly and have the same start and end value (e.g., 0 and 100 % maturity) for all process runs ( Nomikos and MacGregor, 1995 ; Ündey et al, 2002 ; Ündey et al, 2003 ).…”

Industrial Process Monitoring in the Big Data/Industry 4.0 Era: from Detection, to Diagnosis, to Prognosis

“…Detection strength is related with the ability to correctly detect abnormal situations without incurring in excessive false alarms [90]. The figure of merit commonly adopted is the True Positive Rate, TPR, also referred as True Detection Rate, TDR [43,56,91].…”