Things you Wanted to Know about Bias in Evaluations but Never Dared to Think

Abstract

The thrust for evidence-based policymaking has paid little attention to problems of bias. Statistical evidence is more fragile than generally understood, and false positives are all too likely given the incentives of policymakers and academic and professional evaluators. Well-known cognitive biases make bias likely for not dissimilar reasons in qualitative and mixed methods evaluations. What we term delinquent organisational isomorphism promotes purportedly scientific evaluations in inappropriate institutional contexts, intensifying motivated reasoning and avoidance of cognitive dissonance. This leads to states of denial with regard to the validity of much evaluation activity. Independent replications, revisits and restudies, together with codes of ethics that relate to professional integrity, may mitigate these problems.

Peer Review #2 of "A surge of p-values between 0.041 and 0.049 in recent decades (but negative results are increasing rapidly too) (v0.1)"

“…False positives can be caused by questionable statistical practices (called 'p-hacking', 'fiddling', 'spin', or 'data massage') that turn a negative result into a positive result (Boutron et al, 2010;Chan et al, 2014;Dwan et al, 2008;Kirkham et al, 2010;Simmons et al, 2011). Selectively removing particular outliers, undisclosed data dredging, selective stopping, incorrect rounding down of p-values, or trying out various statistical tests and subsequently reporting only the most significant outcomes, are some of the mechanisms that can cause false positives to appear in a publication (e.g., Bakker & Wicherts, 2014;Camfield et al, 2014;Goodman, 2014;Leggett et al, 2013;Simmons et al, 2011;Strube, 2006). Yet another mechanism that could cause spurious positive results is data fabrication.…”

Ethics of Randomized Controlled Trials: Should Economists Care about Equipoise?

“…To some extent, the same biases apply to medical providers. Importantly, single blindness can favor-often subconscious-investigators' self-serving biases (Camfield et al, 2014), hence the advantage of double over single blindness. This argument is particularly relevant for soft or subjective tested outcomes, which are common in social sciences.…”

All You Need is Cable TV?

“…Many economics analyses are underpowered (Ioannidis & Doucouliagos, 2013;Ioannidis et al, 2016), leading to non-trivial risks of false positives (Button et al, 2013), which can be exacerbated by researcher or publication biases (Camfield, Duvendack, & Palmer-Jones, 2014;Maniadis, Tufano, & List, 2014).…”