Many Labs 3: Evaluating participant pool quality across the academic semester via replication

Abstract

The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools (N = 2,696) and with an online sample (N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences-conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects. Word count = 151Keywords: social psychology; cognitive psychology; replication; participant pool; individual differences; sampling effects; situational effects 4 Many Labs 3: Evaluating participant pool quality across the academic semester via replication University participant pools provide access to participants for a great deal of published behavioral research. The typical participant pool consists of undergraduates enrolled in introductory psychology courses that require students to complete some number of experiments over the course of the academic semester. Common variations might include using other courses to recruit participants or making study participation an option for extra credit rather than a pedagogical requirement. Research-intensive universities often have a highly organized participant pool with a participant management system for signing up for studies and assigning credit. Smaller or teaching-oriented institutions often have more informal participant pools that are organized ad hoc each semester or for an individual class.To avoid selection bias based on study content, most participant pools have procedures to avoid disclosing the content or purpose of individual studies during the sign-up process.However, students are usually free to choose the time during the semester that they sign up to complete the studies. This may introduce a selection bias in which data collection on different dates occurs with different kinds of participants, or in different situational circumstances (e.g., the carefree semester beginning versus the exam-stressed semester end).If participant characteristics differ across time during the academic semester, then the results of studies may be moderated by the time at which data collection occurs. Indeed, among behavioral researchers there are widespread intuitions, superstitions, and anecdotes about the "best" time to collect data in order to minimize error and maximize power. It is common, for example, to hear stories of an effect being obtained in the first part of the semester that then "d...

Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

“…Being able to replicate scientific findings is crucial for scientific progress [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15] To what extent can we trust scientific findings? The answer to this question is of fundamental importance…”

Promoting Open Science to Increase the Trustworthiness of Evidence in Special Education

“…Unfortunately, replication studies are seldom conducted (Makel, Plucker, & Hegarty, 2012;Makel & Plucker, 2014). Interestingly, when researchers have conducted replication studies, results have often failed to validate original findings (Camerer et al, 2016, in press;Ebersole et al, 2016;Klein et al, 2014;Open Science Collaboration, 2015). For example, Open Science Collaboration (2015) directly replicated 100 studies in psychology.…”

Modernizing Relationship Therapy through Social Thermoregulation Theory: Evidence, Hypotheses, and Explorations

“…Psychological research has consistently shown that temperature fluctuations (either outside or lab temperature) is causally tied to social behaviors ranging from renting romance movies (Hong and Sun, 2012) to house-purchasing decisions (Van Acker et al, 2016) to basic effects on perception, language use, and memory (IJzerman and Semin, 2009; Schilder et al, 2014; Messer et al, 2017). The effect also works the other way around: if people feel the environment to be socially unpredictable, they perceive temperatures as lower, whereas the reverse is true if people feel psychologically safe (Zhong and Leonardelli, 2008; IJzerman and Semin, 2010; IJzerman et al, 2015b, 2016; Ebersole et al, 2016). The link between psychological safety and thermoregulation extends to consumer behavior: brands that are regarded as more trustworthy induce perceptions of higher temperature, while the degree to which one is affected by temperature determines what one would pay for the brand (IJzerman et al, 2015b).…”

“…Wagemans and IJzerman (2015, unpublished) for example found that peripheral temperature increases, but only if the relationship is communal. Szymkow et al (2013) and IJzerman et al (2015b) find that people estimate temperature higher, but only if the target is trustworthy (and only if lab temperatures are lower; Ebersole et al, 2016; IJzerman et al, 2016). Finally, people’s need to thermoregulate is higher, but again only if they perceive others as trustworthy (i.e., are securely attached; Vergara et al, 2017, unpublished).…”