Exceedance probability analysis: a practical and effective alternative to t-tests

Abstract

This paper presents a practical and effective alternative to the traditional t-tests for (1) comparing a sample or sample mean against a known mean (i.e. one-sample test) and (2) comparing two samples or two sample means (i.e. two-sample test). The proposed method is referred to as exceedance probability (EP) analysis. In a one-sample test, EP is defined as the probability that a sample or sample mean is greater than a known mean. In a two-sample test, EP is defined as the probability that the difference between two samples or between two sample means is greater than a specified value (referred to as probabilistic effect size (PES)). This paper also defines a new statistic called relative mean effect size (RMES). RMES provides a true measure of the scientific significance (not the statistical significance) of the difference between two means. A case study of preference between two manufacturers is presented to demonstrate the effectiveness of the proposed EP analysis, compared with four existing methods: t-tests, common language effect size (CL) analysis, signal content index (SCI) analysis, and Bayesian analysis. Unlike these existing methods that require the assumption of normality, the proposed EP analysis can be performed with any type of distributions. The case study example is examined with a normal distribution model and a raised cosine distribution model. The former is solved with an analytical solution and the latter is solved with a numerical method known as probability domain simulation (PDS).