Effect of the distribution of faults and operating conditions on AFDD performance evaluations

Abstract

Automated fault detection and diagnosis (AFDD) tools are used to identify degradation faults that reduce the performance and life of airconditioning equipment. A recent methodology has been developed to evaluate the performance of AFDD tools. The methodology involves feeding a library of input data to an AFDD protocol and categorizing the results. The current paper describes a study that has been conducted to assess the effect of using various input data sets in the evaluations. These input data sets include different distributions of fault type, fault intensity, and operating temperatures. Case study evaluations of three AFDD protocols in current widespread use are used to demonstrate the effects. The paper shows that evaluation results are sensitive to input data sets, and argues that data sets used in previously published studies should be improved to give higher fidelity evaluations. It concludes that for AFDD performance evaluation to be meaningful, the fault and operating conditions need to be controlled so that they connect to the anticipated deployment conditions. A related conclusion is that it is necessary to use simulation data, rather than laboratory measurement data, to conduct performance evaluation of AFDD.

Building fault detection and diagnostics: Achieved savings, and methods to evaluate algorithm performance

“…Specific to fault diagnostics, while numerous research papers evaluate the performance of individual algorithms (Rossi and Braun 1997, Katipamula et al 1999, Ferrettu et al 2015 it is difficult to draw comparisons or understand the overall state of technology, as each study uses different datasets, test conditions, and metrics. A body of work by Yuill and Braun has explored these concerns, largely with a focus on handheld FDD devices for use with unitary systems (Yuill and Braun 2013, Yuill and Braun 2016, Yuill and Braun 2017. There is a lack of standard methodology and datasets for evaluating the accuracy of FDD technologies that continuously analyze operational data streams from building automation systems and built-up as well as unitary HVAC systems.…”

A performance evaluation framework for building fault detection and diagnosis algorithms

“…Such a procedure would provide a trusted, standard method for validation and comparison of FDD tools at all stages of development, from early-stage research to mature commercial products. Given the wide variety of FDD use cases and competing techniques, establishing a standard evaluation methodology is a daunting challenge [22,23]. Significant progress has been made in establishing FDD test procedures and metrics within both the buildings sector [24,25] and other industries [26,27].…”

“…Building on the Annex 34 work, Reddy [24,31] and Yuill and Braun [23,25,32,33] have contributed significantly to the development of FDD evaluation methodologies for chillers and unitary equipment, respectively. Reddy [24] describes FDD algorithm performance evaluation as one component of a broader evaluation methodology that examines FDD tools' performance, cost, ease of implementation, ease of use, data requirements, training requirements, and applicability to the needs of a particular site or customer.…”

Representing Small Commercial Building Faults in EnergyPlus, Part I: Model Development

“…However, simulated faults can be used indirectly to train data-driven FDD algorithms [7]. Simulation also offers key theoretical advantages over experimental or field data in the evaluation of FDD algorithms: the ground truth is known with greater certainty; the cost of generating evaluation samples is low; evaluation samples can be generated for a wide variety of equipment, building types, environmental conditions, and so on; and evaluation data sets may be constructed without significant gaps or biases [10,12]. However, an accurate simulation of faults is not trivial and rigorous validation is critical if the models are to be trusted [4,10].…”