In this paper, we will discuss the performance, evaluation, and optimization of pattern recognition techniques for applications in system diagnostics. One reason for measuring performance of a diagnostic technique is to clearly quantify it. Another is to compare its performance with that of competing designs. We discuss traditional dichotomous performance measures as well as extensions of these methods to handle multiple classes. We describe a MATLAB toolbox that we have designed to aid developers in rapid testing and optimization. The tool allows the user to select test features, design tests, determine optimal decision thresholds and improve diagnostic performance. The toolbox is demonstrated using modeled engine data. For illustrative purposes, the performances of Partial Least Squares, Principle Component Analysis, Support Vector Machine, and Probabilistic Neural Network data-driven classifiers are compared to that of a model-based classifier developed for a particular engine using modeled data.