A pragmatist philosophy of psychological science and its implications for replication

Gantman, Ana P.; Gomila, Robin; Martínez, Joel E.; Matias, J. Nathan; Paluck, Elizabeth Levy; Starck, Jordan; Wu, Sherry Jueyu; Yaffe, Nechumi Malovicki

doi:10.1017/s0140525x18000626

Cited by 13 publications

(11 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…If one assumes the residuals are simply noise and that block effects (i.e., changes across repetitions) are unimportant for the research question at hand, one can average to obtain relatively residual-less VPC estimates. The former maps the sources of total variance in the data, whereas the latter isolates and maps variance of interest; both may serve to compare estimates across studies or evaluate replications (Gantman et al, 2018). Whichever approach is taken, it is important to report as the residual-less estimates will be greater than those that include intact residuals and may be harder to compare across studies (for related calls toward standardization of analyses see Judd et al, 2012).…”

Section: Analysis Recommendationsmentioning

confidence: 99%

“…Often replications are focused on recapturing the effect sizes of different manipulations. Another criterion by which to judge replications is the extent to which similar influences of participants or stimuli (Gantman et al, 2018), or even unexplained variance (Doherty, Shemberg, Anderson, & Tweney, 2013), are recaptured across studies. The interpretation and possibility of replicating or not replicating an effect size changes if the relative influences of people, stimuli, and unexplained variance underlying the effect change or stay the same.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Quantifying idiosyncratic and shared contributions to judgment

2020

Self Cite

View full text Add to dashboard Cite

Identifying relative idiosyncratic and shared contributions to judgments is a fundamental challenge to the study of human behavior, yet there is no established method for estimating these contributions. Using edge cases of stimuli varying in intrarater reliability and interrater agreement-faces (high on both), objects (high on the former, low on the latter), and complex patterns (low on both)-we showed that variance component analyses (VCAs) accurately captured the psychometric properties of the data (Study 1). Simulations showed that the VCA generalizes to any arbitrary continuous rating and that both sample and stimulus set size affect estimate precision (Study 2). Generally, a minimum of 60 raters and 30 stimuli provided reasonable estimates within our simulations. Furthermore, VCA estimates stabilized given more than two repeated measures, consistent with the finding that both intrarater reliability and interrater agreement increased nonlinearly with repeated measures (Study 3). The VCA provides a rigorous examination of where variance lies in data, can be implemented using mixed models with crossed random effects, and is general enough to be useful in any judgment domain in which agreement and disagreement are important to quantify and in which multiple raters independently rate multiple stimuli.

show abstract

Section: Analysis Recommendationsmentioning

confidence: 99%

mentioning

confidence: 99%

Quantifying idiosyncratic and shared contributions to judgment

2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…To do so, social scientists and policy-makers should collect qualitative and quantitative data to understand how people think about social influence. This is likely to be highly dependent on the context (e.g., school, workplace), which should lead researchers to theorize on the various sources of observers' biases in different settings and improve the general effectiveness of future interventions (Gantman et al, 2018).…”

Section: Discussionmentioning

confidence: 99%

Network insiders and observers: who can identify influential people?

Gomila

Shepherd

Paluck

2020

Behav. Public Policy

Self Cite

View full text Add to dashboard Cite

Identifying influential people within a community to involve in a program is an important strategy of behavioral interventions. How to efficiently identify the most effective individuals is an outstanding question. This paper compares two common strategies: consulting ‘network insiders’ versus ‘network observers’ who have knowledge of but who do not directly participate in the community. Compared to aggregating information from all insiders, asking relatively fewer observers is more cost-effective, but may come at a cost of accuracy. We use data from a large-scale field experiment demonstrating that central students, identified through the aggregated nominations of students (insiders), reduced peer conflict in 56 middle schools. Teachers (observers) also identified students they saw as influential. We compare the causal effect of the two types of nominated students on peer outcomes and the differences between the two types of students. In contrast to the prosocial effects of central students on peer conflict, teacher nominees have no, or even antisocial, influence on their peers’ behaviors. Teachers (observers) generally nominated students with traits salient to them, suggesting that observer roles may systematically bias their perception. We discuss strategies for improving observers’ ability to identify influential individuals in a network as leverage for behavioral change.

show abstract

“…Study 2's sample was collected simultaneously alongside Study 1. In addition to the benefits of replication generally, Study 2 provides a variance-based benchmark for assessing replication (Gantman et al, 2018).…”

Section: Study 2 -Replication Of Studymentioning

confidence: 99%

“…Knowing the directional effects of variables is one way to inform theory. Knowing much they contribute to different variance components and how much variance is left unexplained is an underappreciated tool in theory building (Gantman et al, 2018;Yarkoni, 2019). In this case, one could try to find more important influences on individuals' stances on racism by explaining more perceiver or stimulus variance through additional measures of group characteristics (e.g., system justification; Jost & Banaji, 1994), unique characteristics (e.g., historical knowledge of racism;…”

Section: Shared and Idiosyncratic Contributions To Judgments Of Racismmentioning

confidence: 99%

Analytic racecraft: Race-based averages create illusory group differences in perceptions of racism

Je¹,

El²

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Rising nationalism on the one hand and anti-racist protests on the other have called into question norms surrounding what can be considered racist. How do individuals decide whether anti- immigrant statements are “racist”? This project sought to identify determinants of agreement about what kind of speech is “racist”. Across two studies and one replication (N = 1211) from a nationally representative panel, perceivers rated how “racist” they found social media statements about immigration. The statements were drawn following the announcement of immigration policies in the United States. We introduce variance component analyses to quantify the extent to which judgments of racism are shared or idiosyncratic. We calculated the contributions of perceiver demographics and attitudes (e.g., racial category, age, gender, political leaning, attitudes toward hierarchy and race), statement attributes (e.g., aggressiveness or target of the statement), and context cues (e.g., statement popularity) to measure sources of agreement. While traditional aggregate analyses suggested that perceivers’ judgments about what constitutes racism differed on average between racial categories, the variance analyses consistently identified substantial disagreement, even within racial categories and with varied social norm cues (i.e., number of likes). Agreement occurred when perceivers shared political and racial ideology, and when the statements contained aggressive messages. Judgments of racism may not be shared within predicted social cleavages (e.g., racial categories) in this particular case, because anti-immigrant sentiment is highly politicized. Analyzing variance, as compared to only differences in means, brings new insights to the psychological study of racism, group-based beliefs, and anti-prejudice endeavors.

show abstract

A pragmatist philosophy of psychological science and its implications for replication

Cited by 13 publications

References 10 publications

Quantifying idiosyncratic and shared contributions to judgment

Quantifying idiosyncratic and shared contributions to judgment

Network insiders and observers: who can identify influential people?

Analytic racecraft: Race-based averages create illusory group differences in perceptions of racism

Contact Info

Product

Resources

About