ROC with Cost Pareto Frontier Feature Selection Using Search Methods

Abstract

Cyber-physical systems (CPS) are finding increasing application in many domains. CPS are composed of sensors, actuators, a central decision-making unit, and a network connecting all of these components. The design of CPS involves the selection of these hardware and software components, and this design process could be limited by a cost constraint. This study assumes that the central decision-making unit is a binary classifier, and casts the design problem as a feature selection problem for the binary classifier where each feature has an associated cost. Receiver operating characteristic (ROC) curves are a useful tool for comparing and selecting binary classifiers; however, ROC curves only consider the misclassification cost of the classifier and ignore other costs such as the cost of the features. The authors previously proposed a method called ROC Convex Hull with Cost (ROCCHC) that is used to select ROC optimal classifiers when cost is a factor. ROCCHC extends the widely used ROC Convex Hull (ROCCH) method by combining it with the Pareto analysis for cost optimization. This paper proposes using the ROCCHC analysis as the evaluation function for feature selection search methods without requiring an exhaustive search over the feature space. This analysis is performed on 6 real-world data sets, including a diagnostic cyber-physical system for hydraulic actuators. The ROCCHC analysis is demonstrated using sequential forward and backward search. The results are compared with the ROCCH selection method and a popular Pareto selection method that uses classification accuracy and feature cost.

Making More with Less: Improving Software Testing Outcomes Using a Cross-Project and Cross-Language ML Classifier Based on Cost-Sensitive Training

“…However, a larger range of RC values assigned to FN ({1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 20, 30, 40, 50, 100}) was experimented to broaden the investigation of the effects of the cost-sensitive training in the present domain. Since, as in [76,77], distinct, and a larger number of ML classifiers (RF) models were induced when compared to [40]. The dataset composition and research protocol are depicted in Figure 2.…”

Making more with Less: Improving Software Testing Outcomes Using a Cross-Project and Cross-Language ML Classifier Based on Cost-Sensitive Training

“…However, a larger set of costs assigned to FN ({1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 20, 30, 40, 50, 100}) was experimented to broaden the investigation of the effects of the cost-sensitive training. Since, as in [68,69], distinct, and a larger number of ML classifiers (RF) models were induced when compared to [40]. The dataset composition and research protocol are depicted in Figure 2.…”