Optimization of sample size in controlled experiments: The CLAST rule

Botella, Juan; Ximénez, Carmen; Revuelta, José Luis González de Rivera y; Suero, Manuel

doi:10.3758/bf03192751

Cited by 30 publications

(38 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…vious research has shown that CLAST is more efficient, in terms of sample size and power, than is COAST and that it reflects more realistically the practice of experimental researchers, who obviously are not willing to incorporate an unlimited number of observations into their samples. The CLAST rule has been applied only to the one-tailed t test of mean differences with two matched samples and to the chi-square independence test for twofold contingency tables (Botella et al, 2006). The present work extends previous study on the efficiency of CLAST to multiple group statistical tests.…”

mentioning

confidence: 58%

“…Results of simulation studies indicated that COAST requires about 30% fewer observations than does FSR to achieve the same power. Botella, Ximénez, Revuelta, and Suero (2006) proposed the composite limited adaptive sequential test (CLAST), which is an extension of COAST. Pre-Extending the CLAST sequential rule to one-way ANOVA under group sampling CARMEN XIMÉNEZ AND JAVIER REVUELTA Universidad Autónoma de Madrid, Madrid, Spain Several studies have demonstrated that the fixed-sample stopping rule (FSR), in which the sample size is determined in advance, is less practical and efficient than are sequential-stopping rules.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Extending the CLAST sequential rule to one-way ANOVA under group sampling

Ximénez

Revuelta

2007

Behav Res

Self Cite

View full text Add to dashboard Cite

Several studies have demonstrated that the fixed-sample stopping rule (FSR), in which the sample size is determined in advance, is less practical and efficient than are sequential-stopping rules. The composite limited adaptive sequential test (CLAST) is one such sequential-stopping rule. Previous research has shown that CLAST is more efficient in terms of sample size and power than are the FSR and other sequential rules and that it reflects more realistically the practice of experimental psychology researchers. The CLAST rule has been applied only to the t test of mean differences with two matched samples and to the chi-square independence test for twofold contingency tables. The present work extends previous research on the efficiency of CLAST to multiple group statistical tests. Simulation studies were conducted to test the efficiency of the CLAST rule for the one-way ANOVA for fixed effects models. The ANOVA general test and two linear contrasts of multiple comparisons among treatment means are considered. The article also introduces four rules for allocating N observations to J groups under the general null hypothesis and three allocation rules for the linear contrasts. Results show that the CLAST rule is generally more efficient than the FSR in terms of sample size and power for one-way ANOVA tests. However, the allocation rules vary in their optimality and have a differential impact on sample size and power. Thus, selecting an allocation rule depends on the cost of sampling and the intended precision.

show abstract

mentioning

confidence: 58%

mentioning

confidence: 99%

Extending the CLAST sequential rule to one-way ANOVA under group sampling

Ximénez

Revuelta

2007

Behav Res

Self Cite

View full text Add to dashboard Cite

show abstract

“…The present method is a variation of previous stopping rules originated by Frick (1998) and modified by Botella, Ximénez, Revuelta, and Suero (2006) and Ximénez and Revuelta (2007). These earlier methods did not control alpha at a constant level and were therefore less powerful than the variable-criteria SSR under certain conditions.…”

Section: P Smentioning

confidence: 99%

“…Other SSR approaches have been presented that are effective with large-sample techniques (Botella et al, 2006;Frick, 1998;Ximénez & Revuelta, 2007). Future research could extend the variable-criteria method to smaller effect sizes to assure users of a relatively constant observed alpha.…”

Section: Limitations and Future Researchmentioning

confidence: 99%

The variable-criteria sequential stopping rule: Generality to unequal sample sizes, unequal variances, or to large ANOVAs

Fitts

2010

Behavior Research Methods

View full text Add to dashboard Cite

The variable-criteria sequential stopping rule (SSR) is a method that allows investigators to conduct nullhypothesis significance tests in stages with a few subjects at a time until the null hypothesis is rejected or the experiment is stopped because the obtained p value is so high that it is unlikely that the experiment will succeed (Fitts, 2010). This method controls the rate of Type I errors near the nominal value, such as .05, and maintains excellent power. It allows an investigator to progress intuitively through an experiment without investing large amounts of effort, resources, and subjects all at once. The method also stops experiments early when it is obvious that the null hypothesis is not likely to be rejected, and thus, it saves subjects. For this reason, the method is highly recommended for research when a null hypothesis test is appropriate (Frick, 1996) and when it is imperative to use as few subjects as possible because the subjects are rare or expensive or because the treatments cause pain or distress (e.g., experimental surgeries, tests of pain medication). An SSR is not helpful for creating confidence intervals of small size, because small confidence intervals require large sample sizes. The technique is useful mostly when the researcher is satisfied to know whether an effect exists and, if so, in which direction. My original study (Fitts, 2010) did not validate the use of the variable-criteria SSR with more than four groups in an ANOVA or provide guidance on how to use the method if sample size is lost from one or more groups during the experiment. That is the aim of the present study.The variable-criteria SSR contrasts with the fixed stopping rule, which does not share its benefits. The fixed stopping rule is the customary method for conducting a null hypothesis test, in which the investigator does a power analysis to determine the sample size for the groups, collects data on all subjects, and then analyzes the data with a significance test, such as a t test, once and for all. If the p value is less than alpha (e.g., .05), the investigator rejects the null hypothesis and declares the result significant. If the p value is greater than alpha, little can be concluded. The null hypothesis has not been proven, because the experiment may lack the power to detect an effect of that size. The investigator must not add sample size to the experiment at this point, because the alpha selected for the experiment, such as .05, is constant only for a single test of the null hypothesis. If the investigator adds sample size and compares the new p value with the same criterion of .05, the observed rate of Type I errors will rapidly rise above .05 to unacceptable levels. This leads to an increased publication of unreplicable results.The use of the variable-criteria SSR to control alpha is explained in detail in Fitts (2010). The present method is a variation of previous stopping rules originated by Frick (1998) and modified by Botella, Ximénez, Revuelta, and Suero (2006) and Ximénez and Revuelta (2007). These e...

show abstract

“…Botella, Ximénez, Revuelta, and Suero (2006) refined Frick's technique by adding a maximum sample size beyond which data collection would stop regardless of whether p still lies between α lower and α upper , and Fitts (2010) provides precise values for α lower and α upper for studies with small samples and large effects. Frick (1998), Botella et al (2006), or Fitts (2010 might be recommended for research situations in which an accurate power analysis is infeasible and when additional participants can be readily run individually or in small sets. Our technique might be recommended for situations in which an accurate power analysis can inform the size of N 1 or when multiple rounds of small dataset augmentation is less feasible.…”

Section: Maintaining P < 05 While Augmenting the Datasetmentioning

confidence: 99%

An Ethical Approach to Peeking at Data

Sagarin

Ambler

Lee

2014

Perspect Psychol Sci

109

105

View full text Add to dashboard Cite

When data analyses produce encouraging but nonsignificant results, researchers often respond by collecting more data. This may transform a disappointing dataset into a publishable study, but it does so at the cost of increasing the Type I error rate. How big of a problem is this, and what can we do about it? To answer the first question, we estimate the Type I error inflation based on the initial sample size, the number of participants used to augment the dataset, the critical value for determining significance (typically .05), and the maximum p value within the initial sample such that the dataset would be augmented. With one round of augmentation, Type I error inflation maximizes at .0975 with typical values from .0564 to .0883. To answer the second question, we review methods of adjusting the critical value to allow augmentation while maintaining p < .05, but we note that such methods must be applied a priori. For the common occurrence of post-hoc dataset augmentation, we develop a new statistic, p augmented , that represents the magnitude of the resulting Type I error inflation. We argue that the disclosure of post-hoc dataset augmentation via p augmented elevates such augmentation from a questionable research practice to an ethical research decision.

show abstract

Optimization of sample size in controlled experiments: The CLAST rule

Cited by 30 publications

References 12 publications

Extending the CLAST sequential rule to one-way ANOVA under group sampling

Extending the CLAST sequential rule to one-way ANOVA under group sampling

The variable-criteria sequential stopping rule: Generality to unequal sample sizes, unequal variances, or to large ANOVAs

An Ethical Approach to Peeking at Data

Contact Info

Product

Resources

About