Probabilistic index: an intuitive non‐parametric approach to measuring the size of treatment effects

Abstract

Effect sizes (ES) tell the magnitude of the difference between treatments and, ideally, should tell clinicians how likely their patients will benefit from the treatment. Currently used ES are expressed in statistical rather than in clinically useful terms and may not give clinicians the appropriate information. We restrict our discussion to studies with two groups: one with n patients receiving a new treatment and the other with m patients receiving the usual or no treatment. The standardized mean difference (e.g. Cohen's d) is a well-known index for continuous outcomes. There is some intuitive value to d, but measuring improvement in standard deviations (SD) is a statistical concept that may not help a clinician. How much improvement is a half SD? A more intuitive and simple-to-calculate ES is the probability that the response of a patient given the new treatment (X) is better than the one for a randomly chosen patient given the old or no treatment (Y) (i.e. P(X > Y), larger values meaning better outcomes). This probability has an immediate identity with the area under the curve (AUC) measure in procedures for receiver operator characteristic (ROC) curve comparing responses to two treatments. It also can be easily calculated from the Mann-Whitney U, Wilcoxon, or Kendall tau statistics. We describe the characteristics of an ideal ES. We propose P(X > Y) as an alternative index, summarize its correspondence with well-known non-parametric statistics, compare it to the standardized mean difference index, and illustrate with clinical data.

The Internationalization of Psychology Journals in Brazil: A Bibliometric Examination Based on Four Indices

“…The Mann-Whitney U test was selected. The Mann-Whitney U test is used frequently when data are non-normally distributed (Bergmann, Ludbrook, & Spooren, 2000), and is effective when working with small samples (Acion, Peterson, Temple, & Arndt, 2006). To test Hypotheses 1-4, significant main effects for English text, editorial board makeup, lead author institution, and study style were separately examined with the Mann-Whitney U test set at p < .05 two-tailed.…”

A Comparison of Psychosocial Outcomes in Elderly Alzheimer Caregivers and Noncaregivers

“…Finally, for easier interpretation, outcome variables were summarized into 5 domains: a) Stress, b) mood/affect, c) coping/resource, d) negative coping, and d) health. Mean effect sizes for these domains were presented using the more clinically meaningful Probabilistic Index effect size[P(X>Y)](37). In the present study, P(X>Y) represents the probability that a randomly selected CG has a worse outcome than a NC in the particular outcome.…”

Robust Statistical Methods for Empirical Software Engineering

“…This metric has also been called the area under the receiver curve (AUC) (Kraemer and Kupfer 2006), the measure of stochastic superiority (Â 12 ) (Vargha and Delaney 2000) and the probabilistic index (P (X > Y )) (Acion et al 2006). Arcuri and Briand (2014) recommend using the metric for software engineering data analysis.…”