Randomization Tests or Permutation Tests? A Historical and Terminological Clarification

Abstract

The terms "randomization test" and "permutation test" are sometimes used interchangeably. However, there are both historical and conceptual reasons for making a clear distinction between the two terms. Using a historical perspective, this chapter emphasizes the contributions made by Edwin Pitman and Bernard Welch to arrive at a coherent theory for randomization and permutation tests. From a conceptual perspective, randomization tests are based on random assignment and permutation tests are based on random sampling. The justification of the randomization test derives from the fact that under the null hypothesis of no treatment effect, the random assignment procedure produces a random shuffle of the responses. The justification of the permutation test derives from the fact that under the null hypothesis of identical distributions, all permutations of the responses are equally likely. It is argued that this terminological distinction is crucial for recognizing the assumptions behind each of the tests and for appreciating the validity of the corresponding inferences.

Replicability and randomization test logic in behavior analysis

“…This is the assumption of a random sample, which supposes that the participants in a study had an equally probable chance of being selected from the population of interest, and therefore, are representative of that population. Since most traditional, parametric NHSTs rely on a random sample to make inferences, the “entire model” Greenland et al refer to can be called the random sample model (see Onghena, ). Any interpretation of a p ‐value based on the random sample model depends on whether the experimenter randomly sampled participants from a population.…”

“…The random sample and assignment distinction is also important for terminological reasons, as RTs are often confused with permutation tests and various bootstrapping techniques. RTs are neither types of permutation tests nor types of bootstrapping tests because RTs have their basis in the random assignment model while permutation and bootstrapping tests have their basis in the random sample model (see Onghena, , for a review). These distinctions are important because random assignment is the raison d'être of all RT logic.…”

Resampling Methods: Concept and R Applications

“…Bu durum bir korelasyon veya t testi hesaplamak için pek pratik gözükmemektedir. Onghena (2018), bu iki kavramın hem tarihsel hem de kavramsal açıdan farklılaştığını düşünmektedir. Yazara göre kavramsal bir bakış açısı ile randomizasyon testleri rastgele (seçkisiz) atamaya dayalı iken permutasyon testlerinde rastgele örnekleme söz konusudur.…”

“…Bu çalışmada, belirtilen kavram kargaşasına ve bu kargaşanın çözümüne ilişkin detaylı açıklamalara değinilmeyecektir. Randomizasyon ve permutasyon testlerinin tarihsel gelişimi, kavramsal farklılıkları, Monte Carlo testleri ile ilişkisi gibi konularda detaylı bilgi edinmek isteyen okuyucuların (Onghena, 2018) kaynağına ulaşmaları önerilir. Ayrıca ilgili okuyuculara randomizasyon testlerinde p değeri hesaplanması ve yorumlanması sürecinde dikkat edilmesi gereken boyutlara odaklanan (Onghena & May, 1995) kaynağını incelenmeleri önerilir.…”

Another Look at the Lady Tasting Tea and Differences Between Permutation Tests and Randomisation Tests

“…The statistical literature is very inconsistent in the use of the terms ‘permutation tests’ and ‘randomisation tests’ (Onghena, 2018; Rosenberger et al , 2019). Both terms are often used to refer to tests that involve permutations.…”