In response to recommendations to redefine statistical significance to p ≤ .005, we propose that researchers should transparently report and justify all choices they make when designing a study, including the alpha level.

The notion of language comprehension as mental simulation has become popular in cognitive science. We revisit some of the original empirical evidence for this. Specifically, we attempted to replicate the findings from earlier studies that examined the mental simulation of object orientation, shape, and color, respectively, in sentence-picture verification. For each of these sets of findings, we conducted two web-based replication attempts using Amazon's Mechanical Turk. Our results are mixed. Participants responded faster to pictures that matched the orientation or shape implied by the sentence, replicating the original findings. The effect was larger and stronger for shape than orientation. Participants also responded faster to pictures that matched the color implied by the sentence, whereas the original studies obtained mismatch advantages. We argue that these results support mental simulation theory, show the importance of replication studies, and show the viability of web-based data collection.

Many philosophers of science and methodologists have argued that the ability to repeat studies and obtain similar results is an essential component of science. A finding is elevated from single observation to scientific evidence when the procedures that were used to obtain it can be reproduced and the finding itself can be replicated. Recent replication attempts show that some high profile results--most notably in psychology, but in many other disciplines as well---cannot be replicated consistently. These replication attempts have generated a considerable amount of controversy and the issue of whether direct replications have value has, in particular, proven to be contentious. However, much of this discussion has occurred in published commentaries and social media outlets, resulting in a fragmented discourse. To address the need for an integrative summary, we review various types of replication studies and then discuss the most commonly voiced concerns about direct replication. We provide detailed responses to these concerns and consider different statistical ways to evaluate replications. We conclude there are no theoretical or statistical obstacles to making direct replication a routine aspect of psychological science.

In two experiments, we investigated whether body posture influences people’s estimation of quantities. According to the mental-number-line theory, people mentally represent numbers along a line with smaller numbers on the left and larger numbers on the right. We hypothesized that surreptitiously making people lean to the right or to the left would affect their quantitative estimates. Participants answered estimation questions while standing on a Wii Balance Board. Posture was manipulated within subjects so that participants answered some questions while they leaned slightly to the left, some questions while they leaned slightly to the right, and some questions while they stood upright. Crucially, participants were not aware of this manipulation. Estimates were significantly smaller when participants leaned to the left than when they leaned to the right.

Language comprehenders have been argued to mentally represent the implied orientation of objects. However, compared to the effects of shape, size, and color, the effect of orientation is rather small. We examined a potential explanation for the relatively low magnitude of the orientation effect: object size moderates the orientation effect. Theoretical considerations led us to predict a smaller orientation effect for small objects than for large objects in a sentence-picture verification task. We furthermore investigated whether this pattern generalizes across languages (Chinese, Dutch, and English) and tasks (picture-naming task). The results of the verification task show an orientation effect overall, which is not moderated by object size (contrary to our hypothesis) and language (consistent with our hypothesis). Meanwhile the preregistered picture-picture verification task showed the predicted interaction between object size and orientation effect. We conducted exploratory analyses to address additional questions.