Probability based criteria are proposed for the assessment of cost‐effectiveness of a new treatment compared to a standard treatment when there are multiple effectiveness measures. Depending on the preferences of a policy maker, there are several options to define such criteria. Two such metrics are investigated in detail. One metric is defined as the conditional probability that a new treatment is more effective with respect to the multiple effectiveness measures for patients having lower costs under the new treatment. A second metric is defined as the conditional probability that a new treatment is less costly for patients having greater health benefits under the new treatment. The metrics offer considerable flexibility to a policy maker as thresholds for cost and effectiveness can be incorporated into the metrics. Parametric confidence limits are developed using a percentile bootstrap approach assuming multivariate normality for the joint distribution of the log(cost) and effectiveness measures. A non‐parametric estimation procedure is also developed using the theory of U‐statistics. Numerical results indicate that the proposed confidence limits accurately maintain coverage probabilities. The methodologies are illustrated using a study on the treatment of type two diabetes. Code implementing the proposed methods are provided in the supporting information.