Multiobjective optimization of safety related systems: an application to short-term conflict alert

Abstract

Many safety related and critical systems warn of potentially dangerous events; for example, the Short Term Conflict Alert (STCA) system warns of airspace infractions between aircraft. Although installed with current technology such critical systems may become out of date due to changes in the circumstances in which they function, operational procedures and the regulatory environment. Current practice is to 'tune' by hand the many parameters governing the system in order to optimise the operating point in terms of the true positive and false positive rates, which are frequently associated with highly imbalanced costs.In this paper we cast the tuning of critical systems as a multi-objective optimisation problem. We show how a region of the optimal receiver operating characteristic (ROC) curve may be obtained, permitting the system operators to select the operating point. We apply this methodology to the STCA system, using a multi-objective (1+1)-evolution strategy, showing that we can improve upon the current hand-tuned operating point, as well as providing the salient ROC curve describing the true positive versus false positive trade-off. We also provide results for three-objective optimisation of the alert response time in addition to the true and false positive rates. Additionally, we illustrate the use of bootstrapping for representing evaluation uncertainty on estimated Pareto fronts, where the evaluation of a system is based upon a finite set of representative data.

On the efficient use of uncertainty when performing expensive ROC optimisation

“…In the case of ROC optimisation the variability in the true positive rate, T (x), and false positive rate, F (x), can be quantified using a "stratified" bootstrap sample [4]. We concentrate on the true positive rate, but exactly analogous expressions hold for the false positive rate.…”

“…There are many industrial classification problems that take a significant amount of time to evaluate on the training data provided (e.g. [4] discusses a classifier problem that takes 5 minutes per evaluation). Techniques that can reduce this cost while still leading to good solutions are therefore of significant use.…”

“…Coupled with this the system to be optimised may also be on a specific architecture, and may therefore have limited scope for parallelisation (see e.g. [3], [4] for a problem that demonstrates both these traits). Research has already borne fruit in the development of specialised multiobjective optimisers (MOO) for general time/cost expensive problems (see for instance the recent work of Knowles [5]).…”

Optimizing Decision Trees Using Multi-objective Particle Swarm Optimization

“…The assignment of class can be traced back to a sequence of rules, which make it easy to explain to end users; this also aids tremendously in their application to e.g. safety critical systems [3] or medical applications [4], where 'black box' classifiers have difficulty passing regulatory hurdles due to their opaque processing nature. Additionally their computational complexity is relatively low and are ideally specified for batch processing, meaning they are widely used for real-time classification tasks, and problems requiring a large throughput of data.…”

Safety system design optimisation using a multi-objective genetic algorithm

“…This problem was solved by several methods, with the best results obtained by the SPEA2-based MOGA. Everson and Fieldsend (2006) introduced the multi-objective optimisation based on GAs of safety related and critical systems. This research and others have shown the capability of the multi-objective approach and is the focus of this paper.…”