Unwanted range effects from using within-subject experimental designs.

Abstract

When each man receives a number of conditions in a balanced or random order, an unwanted range effect can sometimes reverse the rank order of the experimental results. With a range effect, responses are influenced by the range of stimuli, by the range of responses used by the man, or by both the range of stimuli and the range of responses. Range effects generally involve a central tendency but not always. There is no way of discovering whether a within-subject design has introduced an unwanted range effect, except by repeating parts of the experiment using a separate-groups design. Textbooks and courses on experimental design in psychology should emphasize the dangers of within-subject designs. Conflicts between experimental results can sometimes be resolved by discarding the results of within-subject designs. Revisions of theory may then be necessary.In a within-subject design, each man receives more than one experimental condition. The order of the conditions is balanced or randomized over the group of men. Edwards (1968, chap. 10) described Latin-square designs, with each man represented by a row or column. Lindquist (1953, chap. 6) called them Treatments X Subject designs. Winer (1971, chap. 4) called them repeated-measures designs on the same men.An experiment with a within-subject design should be regarded as a learning experiment. The men are trained, or partly trained, on all the conditions included in the experiment. The results apply only to men trained like this. The results do not apply to men trained on a different set of experimental conditions, nor to untrained men. In order to produce results which apply to men who have not been trained previously, it is necessary to use separate groups of men for each experimental condition.It is not possible to ensure unbiased results even by restricting each man to the two experimental conditions to be compared. This is because transfer between the two conditions may not be equal in the two directions. The

Binocular integration in line rivalry

“…The existence of a marked change in rivalry onset associated with a change from a within-to a betweensubjects design would have serious implications for the interpretation of existing investigations of phenomenal rivalry . The possibility of such range effects in within-subjects designs of perceptual problems has been emphasized by Poulton (1973).…”

Loci of contextual effects in judgment.

“…Certainly, a procedure should not be advocated based on an uncertain theory that cannot be tested using that procedure (Birnbaum, 1982a;1982b). Consider Poulton's (1973; recommendation to "avoid" contextual effects by asking each subject to make only a single judgment, prom our theoretical viewpoint, this procedure confounds the stimulus and the context: If each subject judged a different "ratio," the judgment could be represented, RIJ = J t j (sj -s,), which shows that a different J tj function would be allowed for each stimulus and standard. This recommendation must be based on the theory that each subject has the same J* function as in Equation 16, a dubious assumption given the results of Figure 14.…”

Incentive effects on risk attitude in small probability prospects

“…Without agreeing with arguments that consider between-subject data the gold standard to which all other results need to be compared (Poulton, 1973), we think that such data may provide additional insights. We thus present some results on the effect of stake-size on risk attitudes for small probability prospects.…”