2018
DOI: 10.1111/ajps.12396
|View full text |Cite
|
Sign up to set email alerts
|

No Harm in Checking: Using Factual Manipulation Checks to Assess Attentiveness in Experiments

Abstract: Manipulation checks are often advisable in experimental studies, yet they rarely appear in practice. This lack of usage may stem from fears of distorting treatment effects and uncertainty regarding which type to use (e.g., instructional manipulation checks [IMCs] or assessments of whether stimuli alter a latent independent variable of interest). Here, we first categorize the main variants and argue that factual manipulation checks (FMCs)-that is, objective questions about key elements of the experiment-can id… Show more

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...
2
1
1

Citation Types

0
118
0
2

Year Published

2019
2019
2024
2024

Publication Types

Select...
7
2

Relationship

1
8

Authors

Journals

citations
Cited by 177 publications
(120 citation statements)
references
References 33 publications
0
118
0
2
Order By: Relevance
“…We included this check immediately after the treatment before the outcome questions. Research indicates that the placement of factual manipulation checks before outcome measures is of little consequence for treatment effects (Kane and Barabas ).…”
mentioning
confidence: 99%
“…We included this check immediately after the treatment before the outcome questions. Research indicates that the placement of factual manipulation checks before outcome measures is of little consequence for treatment effects (Kane and Barabas ).…”
mentioning
confidence: 99%
“…38 Estimated treatment effects in survey experiments are most valid for the sample of respondents who successfully receive treatment (compliers), but less reliable among respondents who are not properly treated (noncompliers). As prior work on experiments suggests, I find heterogeneous treatment effects between compliers and noncompliers (Kane and Barabas 2019). Models interacting treatment assignment with manipulation check success/failure reduce concerns that observed coefficients are biased due to either: (1) distorting estimated treatment effects by aggregating compliers and noncompliers; or (2) restricting the sample by dropping respondents who failed the manipulation check (Aronow et al 2019).…”
Section: Results From An Empirical Analysis Of Lawful Resistancementioning
confidence: 69%
“…T2: Judicial Deference (n = 508)-"Are you aware that the Egyptian/Jordanian Administrative Court recently issued an important ruling in favor of the government, saying that one of the government's new policies was legal?" Next, I assessed whether respondents were properly treated using a factual manipulation check that asked how the court ruled in the question they just read (Kane and Barabas 2019). 34 Subsequently, all respondents were prompted with an identical vignette where a state official invoked eminent domain on their property: 35…”
Section: Demographic Information and Balance Tests About The Sample Amentioning
confidence: 99%
“…is useful for detecting survey satisficers (Kane & Barabas, 2019). We used this question as a robustness check of the results of our experiments:…”
Section: Manipulation Checkmentioning
confidence: 99%